PaperlayerView exact revision on arXiv

Source-linked AI summary

ExploreNet: Learning Where to Explore in Diffusion GRPO

Shuyue Stella Li, Xiaochuang Han, Yulia Tsvetkov, Luke Zettlemoyer

arXiv:2609.38329v1cs.CV

TL;DR

Flow-GRPO explores with isotropic noise even though latent elements differ in how much they change generated images. EXPLORENET learns pre-reward, elementwise noise scales; on Stable Diffusion 3.5 Medium, it improves held-out GenEval2 by 14% over Flow-GRPO and transfers gains across other evaluations.

  • Problem

    Flow-GRPO applies the same noise scale to every latent element, although elements differ in how much perturbations change the generated image.

    Source: §1 · p. 1, §2 · p. 3

  • Method

    EXPLORENET predicts a pre-reward noise scale for each latent element from the current latent, denoising step, and prompt, and trains on each rollout group's reward spread.

    Source: §3.1 · p. 3

  • Results

    14% held-out GenEval2 improvement over Flow-GRPO transfers to two compositional benchmarks and five preference and image-quality models; human raters prefer EXPLORENET 67.2% of the time.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Across the experiments, exploration is learnable, distribution shape outweighs magnitude, and rollout quality is more effective than rollout quantity.

    Source: §4.3 · p. 7, §6 · p. 10

    Channel sensitivity is specific to the SD3.5-M latent space, though a trained EXPLORENET could be reused across rewards and fine-tuning runs of that base model.

    Source: LIMITATIONS · p. 10

Abstract

from arXiv · show

Group-relative RL methods such as Flow-GRPO post-train image generators by exploring with isotropic Gaussian noise added at every denoising step. This noise decides which rollouts the model learns from, yet it perturbs every channel and spatial position of the latent equally. In this paper, we instead show that latent elements differ in how much they change the generated image, so exploration should adapt to these differences. We introduce EXPLORENET to learn an adaptive exploration distribution. EXPLORENET is a policy that predicts a noise scale for every latent element from the current latent, the denoising step, and the prompt, before any reward is observed; it is trained on the reward spread of each rollout group and discarded after training, leaving inference unchanged. On Stable Diffusion 3.5 Medium, EXPLORENET improves held-out GenEval2 by 14% over Flow-GRPO, transfers to two independent compositional benchmarks and five preference and image-quality models, and reaches a 67.2% human preference win-rate. Overall, across our group-relative diffusion RL experiments, we find that exploration is learnable, the shape of the exploration distribution outweighs its magnitude, and rollout quality is more effective than rollout quantity.

1 INTRODUCTION

Flow-GRPO uses isotropic noise at each denoising step to generate the rollouts that guide group-relative learning, but latent elements differ in their image sensitivity. EXPLORENET learns pre-reward elementwise noise scales and improves held-out GenEval2 and broader evaluations.

1 INTRODUCTION

  • Flow-GRPO's isotropic noise guides rollout exploration, but latent elements differ in image sensitivity, so a shared scale allocates noise equally to scene structure and fine detail.Prior work adjusts noise levels by denoising step, while still applying the same scale across each step's channels and spatial positions.Source: §1 · p. 1
  • Post-hoc rollout selection observes rewards before choosing training rollouts and incurs the cost of rollouts it discards; EXPLORENET instead treats stepwise noise as an action rewarded after image scoring.EXPLORENET keeps every rollout it samples for training.Source: §1 · pp. 1–2
  • EXPLORENET predicts a noise scale for every latent element from the current latent, denoising step, and prompt before rewards are observed, then trains jointly with the denoiser on each prompt group's reward spread.The separate policy is removed after training, leaving the trained denoiser's sampler and inference cost unchanged.Source: §1 · pp. 1–2
  • 14% held-out GenEval2 improvement over Flow-GRPO transfers to two compositional benchmarks and five preference and image-quality models; raters prefer EXPLORENET 67.2% of the time.The reported experiments use Stable Diffusion 3.5 Medium trained with GenEval2.Source: §1 · p. 2
  • Controlled comparisons show that exploration-distribution shape outweighs magnitude and that rollout quality is more effective than rollout quantity.The reported comparisons include uniform noise at EXPLORENET's average variance and fewer EXPLORENET rollouts against Flow-GRPO.Source: §1 · p. 2

2 PRELIMINARIES: GROUP-RELATIVE RL FOR FLOW MODELS

Flow-GRPO treats denoising as a group-relative decision process, sampling Gaussian actions around deterministic updates and learning from terminal rewards normalized within each prompt group. Its shared noise scale cannot adapt to differences in how latent elements affect generated images.

2 PRELIMINARIES: GROUP-RELATIVE RL FOR FLOW MODELS

  • Flow-GRPO models denoising as a Markov decision process: the state is the latent, step, and prompt; the action is the next latent; and reward arrives only for the finished image.The action is sampled from a Gaussian centered on the denoiser's deterministic update, supplying exploration and a tractable transition density.Source: §2 · pp. 2–3
  • The transition adds Gaussian noise to the denoiser's mean update, with a step-dependent coefficient and standard-normal noise.This gives Flow-GRPO stochastic transitions despite deterministic flow-matching updates.Source: §2 · p. 3
  • A shared stepwise noise scale treats every prompt, latent, and latent element alike, although latent elements differ in how much perturbations change the generated image.Consequently, the shared scale cannot adapt exploration to those differences.Source: §2 · p. 3
  • Flow-GRPO normalizes terminal rewards within each prompt group, assigns each rollout's terminal advantage to every transition, and uses a clipped denoising ratio.The transition ratio compares the current and data-collecting policies' transition densities.Source: §2 · p. 3

3 EXPLORENET

EXPLORENET treats denoising noise as a second policy action, predicting elementwise scales before reward and learning from prompt-group reward spread.

3 EXPLORENET

  • EXPLORENET is a second policy that predicts a noise scale for each latent element from the current latent, denoising step, and prompt before reward is known.The predicted noise is the policy action; its reward arrives when the finished image is scored.Source: §3 · p. 3

3.1 PREDICTING ELEMENTWISE NOISE SCALES

  • A second SD3.5-M transformer predicts a score per latent element; standardized, clipped log-scales define exploration shape, while magnitude has no separate control.Scores use the current latent, denoising step, and prompt; clipping bounds scales to [e^ℓmin, e^ℓmax].Source: §3.1 · pp. 3–4

3.2 TRAINING FROM REWARD SPREAD

  • Each group’s reward spread is standardized across prompt groups and used as EXPLORENET’s return, giving every sampled noise vector in that group the same advantage.Groups with above-average spread receive positive advantage.Source: §3.2 · p. 4
  • Greatest-spread selection learns fastest and smallest-spread selection slowest, with this ordering holding on eight of nine evaluation metrics.The comparison fixes the rollout pool, update group size, and optimizer.Source: Figure 2 · p. 4, §3.2 · p. 4

3.3 JOINT TRAINING AND INFERENCE

  • Rollout rewards train both policies at different granularities; at inference, EXPLORENET is discarded and the trained denoiser uses the standard flow-matching sampler.The rollout advantage weights the denoiser update, while the group advantage weights EXPLORENET’s update.Source: §3.3 · pp. 4–5

4 EXPERIMENTS

Experiments on SD3.5-M compare EXPLORENET with Flow-GRPO and controls across held-out benchmarks, human judgments, and reduced training budgets. EXPLORENET improves the evaluation suite; ablations indicate that exploration shape and rollout quality matter, and its scales track latent-channel visual sensitivity.

4.1 EXPERIMENTAL SETUP

  • Training uses matched SD3.5-M fine-tuning settings and compares Flow-GRPO, matched-variance isotropic noise, and privileged post-hoc rollout selection.Evaluations include held-out GenEval2, transfer benchmarks and models, and pairwise human preference judgments.Source: §4.1 · p. 5

4.2 EXPLORENET IMPROVES THE TRAINED DENOISER

  • Across three matched seeds, EXPLORENET outperforms Flow-GRPO on every evaluation-suite metric, transfers to unseen benchmarks and quality models, and reaches a 67.2% tie-adjusted human preference rate.Human preference was 45.0% for Flow-GRPO and 37.7% for pretrained SD3.5-M.Source: §4.2 · pp. 5–6, Table 1 · p. 6, Figure 3 · p. 6
  • EXPLORENET trails Flow-GRPO through step 600 on held-out GenEval2, then leads at every later evaluation, with a widening gap and an equal-wall-clock advantage.At equal wall-clock time, EXPLORENET scores 0.532 at update 2400 versus Flow-GRPO’s 0.496 at update 3000.Source: Figure 5 · p. 6, §4.2 · pp. 6–7

4.3 DECOMPOSING MAGNITUDE AND SHAPE OF EXPLORATION

  • At equal average noise variance, uniform noise lowers GenEval2 to 0.437 versus Flow-GRPO’s 0.498, while shaped allocation raises it to 0.567.EXPLORENET also matches privileged selection on GenEval2 (0.567 versus 0.555) and exceeds it on VVRBench-Fast (0.808 versus 0.772) with half the rollouts and reward evaluations.Source: §4.3 · p. 7

4.4 EXPLORENET REMAINS EFFECTIVE WITH FEWER ROLLOUTS AND STEPS

  • With 8 rollouts per prompt, EXPLORENET leads on every metric and scores 0.539 on GenEval2 versus Flow-GRPO’s 0.450 with 8 and 0.498 with 24 rollouts.With 10 denoising steps, it leads on six of nine metrics, including all three compositional benchmarks.Source: §4.4 · p. 7

4.5 EXPLORENET CONCENTRATES NOISE ON VISUALLY INFLUENTIAL CHANNELS

  • Across 16 channels, EXPLORENET’s noise scale and RGB change correlate at r = +0.60; human ratings rise from 1.52 in the bottom scale quartile to 2.23 in the top.On pretrained SD3.5-M, channel-scale and RGB-change rank correlation is positive in 99 of 100 prompts, with mean ρ = +0.42.Source: §4.5 · p. 8

5 RELATED WORK

Prior work explores reward-based post-training, rollout construction, and noise design in diffusion or flow models. EXPLORENET differs by learning how exploration is allocated across latent elements at each denoising step, before rewards are observed.

6 RELATED WORK

  • Reward-based diffusion and flow post-training includes likelihood weighting, policy gradients, differentiable rewards, preference optimization, and KL- or entropy-regularized stochastic control.GRPO adaptations such as Flow-GRPO and DanceGRPO make deterministic samplers stochastic; later methods learn rewards adversarially or optimize group preferences directly.Source: §5 · pp. 8–9
  • Channels assigned larger EXPLORENET noise scales change objects, layout, and background, whereas channels with smaller scales change fine details.Figure 8 labels the average noise multiplier assigned to each channel.Source: Figure 8 · p. 9
  • EXPLORENET learns state-dependent exploration across latent elements, complementing methods that change per-step noise levels, choose which steps explore, or alter reward credit.A per-step schedule sets the noise level at each step; EXPLORENET allocates that noise within the step.Source: §5 · p. 9
  • Unlike methods that spend additional rollouts, perturb conditioning, or act before denoising, EXPLORENET sets noise at every denoising step before rewards are observed.It is trained jointly with the denoiser.Source: §5 · p. 9
  • Prior learned-noise approaches shape noise for data modeling or sample quality, while EXPLORENET learns a diffusion-RL exploration distribution from each rollout group's reward spread and is discarded at inference.The cited flow-model RL analyses study stochasticity with noise fixed by design; deep-RL methods use structured, state-dependent perturbations.Source: §5 · p. 9

6 CONCLUSION

The conclusion presents EXPLORENET as a way to make exploration trainable according to how latent elements affect generated images. Across the experiments, exploration's shape matters more than its magnitude, and rollout quality is more effective than rollout quantity.

6 CONCLUSION

  • Latent elements differ in how much they change generated images, and EXPLORENET makes exploration trainable to adapt to those differences.It predicts a noise scale for each latent element from the current generation state, learns from each rollout group's reward spread jointly with the denoiser, and is discarded after training.Source: §6 · p. 10
  • Across the experiments, the shape of the exploration distribution outweighs its magnitude, and rollout quality is more effective than rollout quantity.The paper also concludes that exploration is learnable.Source: §6 · p. 10
  • The conclusion frames exploration as a diffusion-RL design axis alongside reward design and per-step noise schedules.The final model is sampled as before after EXPLORENET is discarded.Source: §6 · p. 10

LIMITATIONS

The stated limitations and future directions concern the form of the exploration distribution, the rollout statistics used to train it, and the theoretical account of its effects. The discussion also identifies possible extensions and uses within the described settings.

LIMITATIONS

  • The exploration distribution is diagonal Gaussian and models transition noise only; correlated covariances and learning other rollout-variation sources remain possible extensions.The paper gives low-rank or spatially structured covariances as ways to explore coherent changes, and names guidance scale and initial noise as other possible sources.Source: LIMITATIONS · p. 10
  • EXPLORENET trains from each group's reward spread, while spreads of individual reward components or verifier constraint families could support separate distributions for different objectives.Source: LIMITATIONS · p. 10
  • Video latents add a temporal axis over which EXPLORENET could allocate exploration; autoregressive and discrete-diffusion generators could use learned per-position temperatures.Source: LIMITATIONS · p. 10
  • EXPLORENET is a second SD3.5-M transformer, though a lightweight denoiser head could predict the same maps; the learned channel sensitivity is specific to that latent space.The paper suggests reuse across rewards and fine-tuning runs of the same base model, and possible guidance for diverse sampling, localized editing, and inference-time search.Source: LIMITATIONS · p. 10
  • A theoretical account linking latent sensitivity to rollout-group reward spread and explaining when exploration shape outweighs magnitude remains open.Source: LIMITATIONS · p. 10

ETHICS STATEMENT

The paper states that reward-based fine-tuning inherits risks from the models being fine-tuned. Its human evaluation involved volunteer graduate-student annotators.

ETHICS STATEMENT

A.1 HYPERPARAMETERS AND COMPUTE

The appendix specifies the training and evaluation setup, then documents controlled choices for noise-scale parameterization, rollout selection, and an isotropic control. It also defines the benchmark and evaluation protocols used for automatic comparisons.

A.1 HYPERPARAMETERS AND COMPUTE

  • The main setting uses 64 prompts with 24 rollouts each per iteration, while the group-size-8 setting uses 192 prompts and the same 1,536 images.EXPLORENET takes 1.25× as long per optimizer update as Flow-GRPO, but both methods use 25 denoiser evaluations at inference and EXPLORENET is not used.Source: §A.1 · p. 16, Table 3 · p. 17

A.2 NOISE-SCALE PARAMETERIZATION

  • Normalized-and-clipped noise performs best at both reported GenEval2 checkpoints among the compared log-standard-deviation mappings.The comparison uses one training run per row and holds prompts, sampler, group size, and optimization schedule constant.Source: §A.2 · p. 17, Table 4 · p. 17

A.3 REWARD-SPREAD SELECTION

  • Greatest-spread selection is best on eight of nine metrics when training uses eight of 16 rollouts per prompt.It retains the four lowest- and four highest-reward rollouts; the three selection rules share the rollout pool and training schedule, with one run each.Source: §A.3 · p. 17, Table 5 · p. 18

A.4 MATCHED ISOTROPIC CONTROL

  • The matched isotropic control uses a constant transition-noise standard deviation set to match EXPLORENET’s mean predicted variance.Its transition mean is Flow-GRPO’s, and it uses the seed 1 training schedule through step 3000.Source: §A.4 · p. 18

B.1 AUTOMATIC EVALUATION

  • Automatic evaluation uses held-out GenEval2 as the main reward benchmark and also evaluates compositional, preference, and image-quality benchmarks.The primary comparison uses three training seeds and three GenEval2 evaluation passes per seed; other automatic benchmarks use one pass per seed.Source: §B.1 · pp. 18–19

B.2 HUMAN EVALUATION

The human-evaluation appendix describes a blinded, matched-prompt preference study and reports annotator agreement. It also summarizes per-seed results and checks whether gains persist across late checkpoints and prompt resampling.

B.2 HUMAN EVALUATION

  • EXPLORENET wins each pairwise preference comparison reported in Table 6, with ties split equally in the rates.The study uses 72 prompts and three annotators; annotator pairwise exact agreement ranges from 56.9% to 72.7%, with 47.7% of comparisons labeled unanimously.Source: Table 6 · p. 19, §B.2 · p. 19

C.1 PER-SEED RESULTS

  • Table 7 gives per-seed results underlying Table 1 at step 3000; GenEval2 values average three evaluation passes, while each other metric is based on one pass.Source: Table 7 · p. 20

C.2 LATE-TRAINING STABILITY AND PROMPT-LEVEL UNCERTAINTY

  • The GenEval2 gain persists across the final five checkpoints, and prompt-resampling intervals are positive for both GenEval2 and VVRBench-Fast.The stability analysis averages steps 2760 through 3000; the prompt-bootstrap analyses condition on the trained models.Source: Table 8 · p. 20, Table 9 · p. 20, §C.2 · p. 20

C.3 PER-BENCHMARK RESULTS

The benchmark appendix reports EXPLORENET gains across preference, image-quality, and compositional-constraint evaluations. The supplied table captions describe direction of gains but do not provide the underlying values.

Preference and HPSv2.1 benchmarks

  • EXPLORENET scores higher on PickScore and on all four HPSv2.1 prompt styles.Table 10 reports means and sample standard deviations across three training seeds.Source: §C.3 · p. 20, Table 10 · p. 21

Image-quality benchmarks

  • EXPLORENET improves HPSv3, Aesthetic, and ImageReward on every prompt suite in the shared four-suite panel.Table 11 reports results for each metric and prompt suite.Source: §C.3 · p. 20, Table 11 · p. 21

VVRBench-Fast

  • On VVRBench-Fast, EXPLORENET improves grounding, spatial, size, and topology constraint scores.Each score is averaged over prompts containing that constraint family and reported across three training seeds.Source: Table 12 · p. 21

D LATENT CHANNEL ANALYSIS

The channel analysis compares predicted exploration scales with pixel-space and human-rated changes after latent-channel perturbations. Across the reported analyses, larger predicted scales correspond to channels with greater measured change, including before RL training.

Pixel-space intervention analysis

  • The four largest-scale channels average 0.0812 RGB change, versus 0.0570 for the four smallest-scale channels.Across 100 prompts, within-prompt scale–sensitivity rank correlation is positive for every prompt, with minimum Spearman ρ = +0.156.Source: D LATENT CHANNEL ANALYSIS · p. 21, Table 13 · p. 22

Human perceptual evaluation

  • Human-rated change separates the largest- and smallest-scale channel quartiles both before training and after EXPLORENET training.Channels are ranked within each prompt by predicted log standard deviation; annotator ratings are within one scale point in 84.7% of pairwise comparisons.Source: Table 14 · p. 22, D LATENT CHANNEL ANALYSIS · p. 22

Pixel-space change across denoisers

  • The relation between EXPLORENET’s predicted scales and pixel-space sensitivity is also present on the pretrained denoiser, before RL training.The analysis compares channel-mean and within-prompt correlations across denoisers.Source: D LATENT CHANNEL ANALYSIS · p. 22, Table 15 · p. 23
Loading 2609.38329v1…