PaperlayerView exact revision on arXiv

Source-linked AI summary

Manifold-Constrained Initial Noise Optimization for Efficient Generative Model Alignment

Jinho Chang, Jong Chul Ye

arXiv:2610.00365v1cs.LGcs.AIcs.CV

TL;DR

Existing reward-alignment approaches are limited for deterministic few-step generators, while initial-noise methods often require gradients unavailable for black-box rewards. ZeNOVA combines gradient-free guidance and Gaussian-geometry-aware updates; experiments report higher rewards than evaluated zeroth-order baselines, including under reduced query budgets.

  • Problem

    Many alignment approaches assume stochastic, many-step trajectories, and existing initial-noise methods often require gradients through the generator and reward, limiting their use with deterministic few-step models and black-box objectives.

    Source: §1 · p. 1, §3 · p. 3

  • Method

    ZeNOVA performs gradient-free initial-noise alignment with annealed soft-value guidance, manifold-constrained hyperspherical Langevin dynamics, and Metropolis-Hastings jumping.

    Source: §1 · p. 2

  • Results

    Across image and video scenarios, ZeNOVA consistently achieves higher rewards than zeroth-order baselines; reducing its particle count from 64 to 8 lowers ImageReward by only 0.0743.

    Source: §6 · p. 9, §5 · p. 9

  • Takeaways & Limitations

    ZeNOVA retains strong performance with less computational cost and is presented as practical for black-box reward alignment.

    Source: §1 · p. 2, §6 · p. 9

    The paper addresses image generative models, whose outputs may be misleading, biased, or not safe for work and may amplify misinformation, prejudice, or privacy violations.

    Source: ETHICS STATEMENT · p. 10

Abstract

from arXiv · show

Recent advances in distillation and flow-map models have enabled deterministic one- or few-step generation for high-quality data, facilitating a new branch of reward alignment approaches that directly optimize the initial noise from a Gaussian distribution. However, most existing initial-noise optimization methods rely on first-order gradient information, which is either inapplicable or suffers from instability and inefficiency in black-box reward scenarios. Here, we introduce ZeNOVA, a stable and efficient initial noise alignment method in a gradient-free manner. Specifically, we address existing algorithms' major challenge in black-box scenarios through annealed soft-value guidance, manifold-constrained hyperspherical Langevin dynamics, and Metropolis-Hastings jumping. Extensive experiments on image and video generative models show that ZeNOVA outperforms all evaluated zeroth-order baselines by optimizing the initial noise toward higher rewards substantially more stably while exploiting the geometry of the Gaussian prior, demonstrating its practical applicability to various black-box reward alignment.

1 INTRODUCTION

Reward alignment methods often rely on stochastic, many-step sampling, limiting their transfer to deterministic few-step generators. ZeNOVA instead optimizes initial noise with gradient-free black-box guidance while accounting for the Gaussian prior geometry.

1 INTRODUCTION

  • Earlier alignment approaches rely on stochastic trajectories with many intermediate steps, making their extension to deterministic generative models limited or not guaranteed to behave as designed.Distilled and flow-map models can generate in one or a few steps, reducing the scope for intermediate control.Source: §1 · p. 1
  • Initial-noise optimization changes a pretrained deterministic generator’s Gaussian input rather than its parameters, but most existing approaches differentiate through both the generator and reward computation.That requirement limits use with non-differentiable or black-box rewards and can add substantial computational overhead even when gradients are available.Source: §1 · pp. 1–2
  • ZeNOVA combines annealed soft-value guidance, hyperspherical updates, and Metropolis-Hastings jumping for stable local refinement and global exploration under black-box rewards.Its updates respect the Gaussian prior geometry.Source: Figure 1 · p. 2, §1 · p. 2
  • ZeNOVA uses only generator and reward-model forward evaluations, and experiments on image and video models report higher rewards than evaluated zeroth-order baselines with more stable optimization.The approach is presented as practical for black-box reward alignment.Source: §1 · p. 2

2 PRELIMINARIES

The preliminaries frame alignment as balancing reward against deviation from a pretrained reference, then express the target as a reward-tilted distribution over initial noise. They motivate preserving the Gaussian prior because unconstrained reward maximization can collapse toward a few regions and degrade image quality.

2 PRELIMINARIES

  • Entropy-regularized reinforcement learning balances expected terminal reward against divergence from the reference process using a positive parameter λ.For diffusion and flow-matching models, trajectory-level KL can decompose into local policy divergences along the sampling trajectory.Source: §2 · p. 2
  • The optimal policy induces a marginal distribution whose soft value function represents expected reward and KL penalty over the optimal policy from state x_t.The difference between the optimal and reference models is proportional to the soft-value gradient.Source: §2 · p. 2
  • Path-Integral Control expresses the optimal policy and soft value through weighted samples of reference dynamics, using terminal desirability to weight sampled noise increments.The control can be estimated from reference-model samples and terminal reward evaluations alone.Source: §2 · pp. 2–3
  • The reward-tilted noise distribution combines a Gaussian input-noise prior with reward, while unconstrained maximization over all noise values risks mode collapse and image-quality degradation.When reward is differentiable with respect to noise, its score is tractable and Langevin dynamics can have the reward-tilted distribution as its stationary distribution.Source: §2 · p. 3

3 METHOD

Section 3 presents ZeNOVA as a response to unreliable direct zeroth-order Langevin optimization, combining annealed soft-value guidance, hyperspherical updates, and Metropolis–Hastings jumps in a forward-evaluation-only procedure.

3.1 CHALLENGES FOR BLACK-BOX OBJECTIVES.

  • Naive zeroth-order Langevin replaces the reward gradient with a Gaussian-search estimate, but flat rewards erase its signal, few-particle estimates destabilize high-dimensional updates, and incremental moves can miss separated modes.The estimator's inaccuracy can also drive the noise out of distribution and cause sample collapse.Source: §3.1 · pp. 3–4

3.2 ZENOVA: NOISE OPTIMIZATION FOR BLACK-BOX REWARDS WITH VALUE ANNEALING

  • ZeNOVA tilts the Gaussian prior by a soft value, whose Gaussian-averaged reward is continuous and whose decreasing horizon schedule moves from broad exploration toward the reward-tilted target.Larger horizons aggregate reward information across a broad neighborhood; smaller horizons bring the intermediate target closer to the original reward tilt for local refinement.Source: §3.2 · pp. 4–5
  • A Gaussian score identity estimates the soft-value gradient from reward evaluations, avoiding differentiation through the reward function.The estimator uses N reward evaluations.Source: §3.2 · p. 5
  • Hyperspherical Langevin updates separate angular motion on the sphere from radial motion governed by the χ_d reference prior, addressing noisy guidance’s norm drift without freezing the radius.The radial update remains stochastic and reward-adaptive, using an analytic one-dimensional prior score rather than an explicit norm constraint.Source: §3.2 · p. 5
  • Interleaved Metropolis–Hastings proposals from the prior provide occasional global moves between reward modes; standalone MH is inefficient, so jumps are used early before local refinement.After the early jumps, annealed soft-value and hyperspherical Langevin updates perform coarse-to-fine refinement.Source: §3.2 · pp. 5–6
  • The combined ZeNOVA procedure requires only forward evaluations of the generator and reward model, with particle count N and iteration count K governing inference-time computation.Increasing N improves the Monte Carlo soft-value-gradient estimate; increasing K allows more gradual traversal of the interpolation.Source: §3.2 · p. 6

4 EXPERIMENTS

ZeNOVA improves black-box text-to-image alignment over the evaluated zeroth-order baselines and also improves VideoAlign scores on rCM. The experiments assess reward alignment across generators and tasks, alongside image quality and a toy sampling problem.

Toy Experiment

  • In a 2-D reward-aligned sampling problem, hyperspherical Langevin updates reproduce the ground-truth distribution, unlike the tested alternatives.The compared alternatives either concentrate in partial high-reward regions or produce artifacts such as samples far along reward bands.Source: §4 · p. 6

Text-to-Image Generation

  • The text-to-image evaluation fixes generator parameters and tests five tasks, including three with black-box rewards, against uncontrolled, zeroth-order, and gradient-based baselines.The evaluated tasks are ImageReward, PickScore, GenEval, object counting, and OCR.Source: §4 · p. 6
  • +0.046 on GenEval and +0.088 on OCR over the best zeroth-order baseline; counting MAE falls by around 40%.Across both text-to-image models, ZeNOVA improves over uncontrolled generation, Best-of-K, and every reported zeroth-order baseline on all five task metrics.Source: §4 · pp. 6–7, Table 1 · p. 7
  • ZeNOVA achieves the highest Aesthetic score across varied scenarios, while some unstable zeroth- and first-order methods show significant drops.Aesthetic score is reported separately to assess how task-specific alignment affects visual quality.Source: §4 · pp. 6, 8

Text-to-Video Generation

  • ZeNOVA improves over uncontrolled generation in all four rCM optimization settings, including individual and combined reward objectives.The experiments apply VideoAlign scores for visual quality, motion quality, and text alignment without modifying the pretrained generator.Source: §4 · p. 8, Table 2 · p. 8

5 DISCUSSION

ZeNOVA maintains strong alignment with fewer reward queries and achieves a stronger reward-diversity trade-off. Ablations report contributions from all three components, with annealed soft-value guidance particularly useful for OCR.

Effect of Particle Count per Iteration

  • With 8 particles, ZeNOVA outperforms all three baselines with 64; reducing its particle count from 64 to 8 lowers ImageReward by only 0.0743.Across reported particle counts, ZeNOVA has the highest reward; Figure 6 compares it with Cartesian Langevin, exponential-map updates, and NES at fixed K = 25.Source: §5 · pp. 8–9, Figure 6 · p. 9

Reward-Diversity Trade-off

  • As 1/λ increases, average reward rises while diversity falls; ZeNOVA has a more diverse output at a fixed reward level.Figure 7 plots the PickScore–Vendi diversity Pareto curve for different methods at three guidance scales.Source: §5 · p. 9, Figure 7 · p. 9

Ablation Study

  • All three ZeNOVA components contribute to performance and stability, while annealed soft-value guidance yields substantial OCR gains but marginal or no gains on ImageReward and PickScore.The authors attribute the OCR result to the value function providing a more global and meaningful reward-gradient signal for that more challenging black-box reward.Source: §5 · p. 9, Table 3 · p. 9

6 CONCLUSION

ZeNOVA is presented as a stable, efficient initial-noise optimization framework that achieves higher rewards than zeroth-order baselines with less computational cost. The paper also notes potential risks from image-generative-model outputs and describes its reproducibility materials.

6 CONCLUSION

  • ZeNOVA combines initial-noise optimization with a reward-tilted sampling objective and is reported to achieve higher rewards than zeroth-order baselines with less computational cost.The authors describe experiments across various scenarios.Source: §6 · p. 9
  • The paper identifies misleading, biased, or not-safe-for-work visual content as potential risks of image generative models.The stated concerns include misinformation, prejudice, and privacy violations.Source: ETHICS STATEMENT · p. 10
  • The authors state that theoretical assumptions and proofs are included in the Appendix and that experimental datasets are open access with data preparation described there.Source: REPRODUCIBILITY STATEMENT · p. 10

A THEORETICAL RESULTS

The theoretical appendix presents a soft-value distribution interpolation result and a Metropolis-Hastings result for sampling from the reward-tilted distribution.

A.1 PROOF OF PROPOSITION 1

  • The first proof uses the soft value function and reversibility of the OU kernel with respect to the prior in deriving its interpolation result.Source: §A.1 · p. 14

A.2 PROOF OF PROPOSITION 2

  • An independent Gaussian-prior proposal with reward-based acceptance probability yields a Metropolis-Hastings chain converging to the reward-tilted distribution.The acceptance probability is min{1, exp((R(z′) − R(z))/λ)}.Source: §A.2 · p. 14

A.3 DERIVATION OF ZEROTH-ORDER VALUE GRADIENT ESTIMATOR

The derivation relates optimal control in an OU reference process to soft values and rewrites its estimator using terminal Gaussian samples. The resulting expression avoids sampling a full OU trajectory or initial Brownian increment.

A.3 DERIVATION OF ZEROTH-ORDER VALUE GRADIENT ESTIMATOR

  • In the OU reference process, stochastic optimal control relates the injected optimal control to a soft-value objective, with expectations under the uncontrolled process and reward at the terminal state.Source: §A.3 · p. 15
  • The derivation rewrites the numerator involving terminal reward and the initial Brownian increment using the terminal Gaussian sample and a conditional-mean formula for jointly Gaussian variables.Source: §A.3 · p. 15
  • The resulting expression depends only on a Gaussian perturbation and its terminal reward, so it requires neither a full OU trajectory nor an initial Brownian increment sample.Source: §A.3 · p. 16

A.4 DERIVATION OF HYPERSPHERICAL LANGEVIN DYNAMICS

The hyperspherical dynamics separate angular motion on the unit sphere from reward-adaptive radial motion. The algorithm combines these updates with annealed soft-value guidance and Metropolis-Hastings jumps, and the appendix describes zeroth-order baselines.

A.4 DERIVATION OF HYPERSPHERICAL LANGEVIN DYNAMICS

  • The method decomposes Gaussian noise into radius and direction, then treats angular motion on the sphere separately from radial motion controlling distance from the origin.The radial update retains the exact Gaussian radial bias while adding reward-induced radial preference.Source: §A.4 · pp. 16–17

B.2 IMPLEMENTATION OF ZEROTH-ORDER BASELINES

  • The described zeroth-order baselines include Langevin and exponential-map updates; NES uses an isotropic Gaussian search distribution with fixed σ and optimizes its mean.Source: §B.2 · p. 17

B.1 DETAILED ALGORITHM OF ZENOVA

  • ZeNOVA’s detailed algorithm combines annealed soft-value guidance, hyperspherical Langevin updates, and independent Metropolis-Hastings jumps using Gaussian-prior proposals and reward-based acceptance.The reciprocal temperature 1/λ acts as an effective reward-guidance scale.Source: §B.1 · p. 17, §B.2 · p. 18

B.3 TEXT-TO-IMAGE GENERATION

The text-to-image and text-to-video experiments specify their generators, reward tasks, prompts, and particle-evaluation budgets.

B.3 TEXT-TO-IMAGE GENERATION

  • Text-to-image experiments use SDXL-Turbo and DMD2, evaluating preference, text alignment, counting, and OCR rewards.Counting outputs are rounded to whole numbers to create a black-box reward setting.Source: §B.3 · p. 18
  • Prompts come from task-specific sources, including 200 randomly selected GenEval test prompts.Counting uses quantity-aware prompts; ImageReward uses HPDv2, while PickScore and OCR use Pick-a-Pic prompts.Source: §B.3 · p. 18
  • Text-to-image zeroth-order methods use 400 or 800 reward evaluations; ZeNOVA uses four MH-jump evaluations per update and task-specific guidance and horizon settings.The settings include 16 particles per update, 25 or 50 iterations, guidance scales of 100 for counting and 1000 otherwise, and geometrically interpolated horizons.Source: §B.3 · pp. 18–19

B.4 TEXT-TO-VIDEO GENERATION

  • Text-to-video experiments use the few-step rCM generator and VideoAlign’s visual-quality, motion-quality, and text-alignment rewards on 30 VidProM prompts.The four objectives are each reward individually or their sum.Source: §B.4 · p. 19
  • Each video reward setting uses 32 particles for 15 iterations, totaling 480 particle reward evaluations.Other hyperparameters match the text-to-image experiments.Source: §B.4 · p. 19

C ADDITIONAL RESULTS

The additional results provide qualitative examples of ZeNOVA outputs across image and video generators and several reward signals.

C ADDITIONAL RESULTS

Related papers

Loading 2610.00365v1…