Learning to Sample Effective and Diverse Prompts for Text-to-Image Generation

• Direct diffusion-model fine-tuning is inaccessible for closed-source models and increasingly costly as model sizes grow, motivating parameter-free prompt adaptation.Source: §1 · pp. 1–2
• Promptist’s reward-maximizing RL concentrates on narrow high-reward regions, producing similar postfixes and limiting generalization across prompts and diffusion models.Source: Figure 1 · p. 1, §1 · p. 2
• PAG uses GFlowNets to sample adapted prompts from an unnormalized reward distribution, targeting both effectiveness and diversity.Source: §1 · p. 2
• PAG addresses naive GFlowNet mode collapse through flow reactivation, reward-prioritized sampling, and progressive reward decomposition for more precise credit assignment.Source: §1 · p. 2
• Experiments report effective and diverse adapted prompts, robustness across reward functions, and zero-shot transfer across text-to-image diffusion models.Source: §1 · p. 2

2.1. Prompt Adaptation

• Prompt adaptation samples an improved prompt y from a language model conditioned on initial prompt x, then uses a diffusion model to generate images from y.Source: §2.1 · p. 2
• The adaptation reward combines image-aesthetic improvement and relevance to the initial prompt, while a KL term keeps generated prompts close to natural language.Source: §2.1 · p. 2

2.2. Generative Flow Networks (GFlowNets)

• GFlowNets sample compositional terminal objects proportionally to an unnormalized reward by generating trajectories through a directed acyclic state-action graph.Source: §2.2 · pp. 2–3
• GFlowNet forward policies can be parameterized by neural networks and trained with trajectory-balance or detailed-balance objectives involving backward policies or state flows.Source: §2.2 · p. 3
• Forward-looking training extends rewards to intermediate states because terminal-only rewards create delayed credit assignment.Source: §2.2 · p. 3

2.3. Dormant Neuron Phenomenon

• Dormant-neuron analysis quantifies neural plasticity by tracking activation scores across hidden units during training.Source: §2.3 · p. 3

3.1. Problem Formulation

• GFlowNets learn a stochastic policy that samples adapted prompts proportional to their rewards, preserving both effectiveness and diversity.This reframes prompt adaptation as sampling from an unnormalized reward density rather than maximizing reward alone.Source: §3.1 · p. 3
• Naively fine-tuning GFlowNets causes mode collapse: prompt diversity decreases as dormant-neuron proportions increase during training.The observed loss of neural plasticity constrains exploration of diverse regions in the prompt space.Source: §3.1 · pp. 3–4, Figure 3 · p. 4
• Terminal-only rewards make sequential credit assignment inefficient, encouraging conservative exploitation of discovered high-reward patterns instead of diverse alternatives.The resulting loss of plasticity and weak intermediate feedback form a self-reinforcing cycle that produces increasingly similar prompts.Source: §3.1 · p. 4

3.2. Proposed Method

• PAG combines flow reactivation, reward-prioritized sampling, and decomposed rewards to retain expressivity, preserve high-quality knowledge, and provide fine-grained learning signals.These complementary mechanisms are designed to resist mode collapse while maintaining training stability.Source: §3.2 · p. 4
• Flow reactivation periodically reinitializes only the last flow-function layer, while leaving the forward policy unchanged to avoid destabilizing prompt sampling.The targeted reset maintains neural plasticity without directly perturbing the policy that interacts with the environment.Source: §3.2 · pp. 4–5
• Reward-prioritized replay samples prompts with probabilities proportional to reward, accelerating recovery of high-quality knowledge after resets and reducing forgetting.This approach addresses the extensive re-exploration that full reset strategies would require in the vast token space.Source: §3.2 · p. 5
• Progressive reward decomposition extends terminal rewards to intermediate states, enabling token-level credit assignment and increasing prompt diversity.The method exploits the step-wise decomposition of the likelihood term and incorporates local signals into the training loss.Source: §3.2 · p. 5
• The training algorithm alternates between on-policy prompt generation and reward-prioritized replay, updates the policy and flow function, and periodically resets the flow layer.Its inputs include an initial prompt dataset, pretrained language model, training rounds, reset period, and batch size.Source: §3.2 · p. 5

4.2. Performance Comparison

• PAG consistently surpasses all baselines in both reward and diversity across various input types.Vanilla GFlowNets generally achieves higher rewards but suffers mode collapse, while Promptist improves reward with limited diversity.Source: §4.2 · p. 6, Table 1 · p. 6
• On the challenging ChatGPT dataset, PAG maintains robust performance while Promptist and vanilla GFlowNets suffer severe mode collapse.PAG also produces adjectives and detailed descriptions rather than only recurring postfixes.Source: §4.2 · p. 6, Figure 5 · p. 7

4.3. Robustness across Different Reward Functions

• PAG achieves state-of-the-art reward and diversity across ImageReward and HPSv2, demonstrating robustness across multiple reward functions.The evaluation uses COCO while maintaining the primary training procedure.Source: §4.3 · pp. 6–7

4.4. Transferability to Different Text-to-Image Diffusion Models

• PAG consistently generates high-rewarding images on SD v1.5, SSD-1B, SDXL-Turbo, and SD3 after training with SD v1.4.The reported gap over baselines is especially significant on SD3.Source: §4.4 · p. 7, Table 2 · p. 7

4.5. Comparison with Fine-tuning Text-to-Image Diffusion Models

• PAG achieves competitive aesthetic quality with DDPO while generating more diverse images than DDPO’s converged styles.This comparison supports prompt adaptation as a potential alternative and complement to direct diffusion-model fine-tuning.Source: §4.5 · p. 7, Figure 6 · p. 8

4.6. Ablation Studies

• Flow reactivation, reward-prioritized sampling, and reward decomposition each substantially contribute to overall performance on COCO and ChatGPT.Removing reward-prioritized sampling causes significant degradation, while reward decomposition improves both reward and diversity.Source: §4.6 · pp. 7–8, Table 3 · p. 8
• Flow reactivation lowers dormant-neuron rates and prevents the prompt-policy mode collapse observed without reactivation.The mechanism maintains flow expressivity, contributing to performance and diversity improvements.Source: §4.6 · p. 8, Figure 7 · p. 8

Aligning Diffusion Models

• Direct diffusion-model fine-tuning requires model-parameter access and model-specific retraining, limiting use with closed-source or rapidly growing models.PAG instead fine-tunes language models for prompt adaptation, enabling transfer across text-to-image models.Source: §5 · p. 8

Prompt Adaptation for Text-to-Image Models

• Promptist frames prompt adaptation as reinforcement learning, but its reward-maximizing policy tends toward narrow, deterministic prompt outputs.PAG targets effective and diverse prompts through a different sampling objective.Source: §5 · p. 8

GFlowNet Fine-Tuning

• GFlowNets sample compositional objects proportional to unnormalized density, yet prior work largely overlooked plasticity loss during fine-tuning and often suffered mode collapse.These limitations motivate PAG’s focus on diverse conditional prompt generation.Source: §5 · pp. 8–9
• PAG uses GFlowNets to fine-tune language models for effective and diverse prompt generation while addressing mode collapse in text-to-image prompt adaptation.The approach applies probabilistic inference to conditional prompt generation rather than relying on reward maximization alone.Source: §5 · pp. 8–9

6. Conclusion

• PAG addresses mode collapse in prompt adaptation through GFlowNets-based probabilistic inference and reports effective, diverse prompt sampling.The conclusion identifies plasticity loss during sequential reward learning as a critical problem that PAG targets.Source: §6 · p. 9

A. Illustrations of RL and GFlowNets

• GFlowNets learn stochastic policies with p(x) ∝ R(x), whereas RL maximizes rewards; reward-proportional sampling can capture high-quality, multimodal candidates.The distinction is especially relevant when reward proxies are inaccurate or reward-maximizing RL harms generalizability.Source: A. Illustrations of RL and GFlowNets · p. 13, Figure 8 · p. 13

B.1. Data Preparation

• The experiments prepare training and evaluation data by following Promptist and adding brief image descriptions generated through ChatGPT.Example descriptions include simple scenes such as a bird on a branch and a boat on a lake.Source: §B.1 · p. 13

B.2. Baselines

• Promptist’s rule-based comparison appends frequently used postfixes, operationalizing the observation that Promptist often generates similar optimization postfixes.The examples contain recurring style and artist descriptors.Source: §B.2 · p. 13
• The baselines include supervised fine-tuning, PPO-based Promptist, a rule-based postfix method, vanilla GFlowNets, and DPO-Diff.The vanilla GFlowNets baseline uses VarGrad because directly estimating a conditional partition function with the TB objective is unstable.Source: §B.2 · pp. 13–14

B.3. Training and Evaluation

• Evaluation generates 16 prompts per initial prompt, computes reward from three images per prompt, and averages the maximum prompt score across initial prompts.Diversity is the average pairwise cosine distance between MiniLMv2 prompt embeddings, reported with four-seed mean and standard deviation.Source: §B.3 · p. 14

C. Extended Main Results

• The appendix provides hyperparameters, extended ablations, and additional analysis of the main experimental results.Table 4 lists training hyperparameters, while the extended materials include ablations and result discussions.Source: Table 4 · p. 14, Figure 9 · p. 15, §B.5 · p. 15

B.4. Robustness across Different Reward Functions

• Robustness is evaluated with ImageReward and HPSv2, which score text-image preference using BLIP-based and CLIP-based preference models, respectively.ImageReward combines image and text embeddings with cross attention, while HPSv2 is fine-tuned on HPDv2 human-preference data.Source: §B.4 · p. 14

B.5. Transferability to Different Text-to-Image Diffusion Models

• Transferability is tested on SD v1.5, SSD-1B, SDXL-Turbo, and SD3 with scheduler and guidance-scale settings adjusted to model compatibility.SDXL-Turbo uses guidance scale 0, while the other models use 7.5; SDXL-Turbo and SD3 use PNDM with 20 inference steps.Source: §B.5 · p. 14

C.1. Main Results

• DPO-Diff shows modest reward improvements compared with other baselines, while prompt diversity is not meaningful because its initial prompt remains fixed.The comparison appears in the main-results table, whose Aes score measures aesthetic-quality improvement over the original input.Source: §C.2 · p. 15, §C.1 · p. 15

C.2. Discussion

• DPO-Diff effectively improves CLIP scores by optimizing negative prompts but shows limited capability in improving aesthetic scores.This separates its text-image relevance gains from its weaker aesthetic-quality improvement.Source: §C.2 · p. 15

D. Extended Ablation Studies

• Extended ablation studies complement the main text by examining additional components of the proposed framework.These studies are included because of space limitations in the main paper.Source: D. Extended Ablation Studies · p. 15

D.1. Analysis on β

• β = 0.05 is the default inverse temperature balancing task reward with reference-language-model likelihood.β controls the balance between r(x, y) and p_ref(y|x).Source: §D.1 · p. 15
• Too-small β produces mode collapse and poor performance, while the tested β values otherwise show no significant performance differences.The tested values are {0.01, 0.05, 0.1, 0.2}.Source: §D.1 · p. 15

E. Comparison with Directly Fine-tuning Diffusion Models

• Figure 10 compares DPOK and PAG by reporting the aesthetic scores of their images, with the reported scores highlighted in bold.Source: Figure 10 · p. 16

D.2. Analysis on M

• M = 2000 achieves the best reward and diversity, whereas frequent reactivation at M = 1000 significantly reduces performance by missing high-reward regions.Rare resets can cause mode collapse, while M = 2000 reactivates the flow function four times during training.Source: §D.2 · pp. 15–16

D.3. Analysis on learning rate of flow function

• γ = 1 × 10^-4 gives the best reward and diversity, while matching the policy learning rate causes poor performance.The default flow-function learning rate is higher than the policy rate to accelerate recovery after flow reactivation.Source: §D.3 · p. 16

D.4. Analysis on flow reactivation scheme

• Resetting only the last flow-function layer outperforms resetting the whole flow layer or policy parameters.Whole-layer resets fail to recover high-scoring regions, while policy resets perform poorly because the policy controls online-sample acquisition.Source: §D.4 · p. 16

D.5. Analysis on Diversity of Images

• PAG produces high image diversity relative to baselines, measured by average pairwise distances between InceptionV3 feature vectors.The evaluation is summarized in Table 6.Source: §D.5 · p. 16, Table 6 · p. 16

E. Comparison with Directly Fine-tuning Diffusion Models

• The direct-fine-tuning comparison evaluates aesthetic quality using prompts from 45 common animals under the DDPO and DPOK experimental setup.The comparison concerns directly fine-tuning diffusion models rather than adapting prompts.Source: E. Comparison with Directly… · p. 16

E.1. Experiment Setup

• The experiment fine-tunes diffusion models with aesthetic quality as the reward and uses prompts from 45 common animals, following DDPO and DPOK setups.The setup is designed for comparison with direct diffusion-model fine-tuning.Source: §E.1 · p. 16

E.2. Comparison with DPOK

• PAG generates diverse and high-quality images, while DPOK converges to similar styles.This comparison is shown in Figure 10.Source: §E.2 · p. 16

F. Additional Visualizations

• Figures 11–12 provide additional Stable Diffusion v1.4 images generated from optimized prompts using the same seed.The fixed seed isolates the effect of prompt adaptation.Source: Figure 11 · p. 18, Figure 12 · p. 18
• Figure 13 visualizes images generated with prompts fine-tuned using different reward functions under the same-seed protocol.The visualization focuses on the effect of prompt adaptation.Source: Figure 13 · p. 19
• Figure 14 visualizes images generated with different text-to-image diffusion models using the same seed.The accompanying discussion summarizes transferability across diffusion models.Source: F. Additional Visualizations · p. 17, Figure 14 · p. 19