T-DIG: Text-Conditioned Diversity for Image Generation

1 INTRODUCTION

• Users request distinct kinds of variation, while prior methods either maximize generic diversity or target a fixed variation type.Examples include changing camera angle, activities, ethnicity, dog breeds, or story endings.Source: §1 · pp. 1–2
• T-DIG formulates diversity as generating K images that vary along a free-form text axis while following the base prompt.Its LLM policy writes axis-conditioned prompts or edit instructions for a frozen T2I model, and training rewards requested-axis diversity while penalizing lost alignment.Source: §1 · pp. 2–3
• Across four open-weight T2I models, T-DIG raises requested-axis diversity 2.3× on average over i.i.d. sampling, with less than 2% alignment loss.The gain reaches 3.1×; human raters prefer T-DIG’s requested-axis diversity over each baseline in 64–96% of instances across the studied models.Source: §1 · p. 2
• The interactive framework lets users request, control, and iteratively refine variations in natural language through generation or editing.Users can specify prompt-specific axes that a fixed taxonomy would not cover.Source: §1 · p. 2
• The paper releases human-grounded training and validation data and a test set from real user interactions, alongside a human rating study against three baselines.Source: §1 · p. 3

2 FRAMEWORK DESIGN

• Users begin with K independent samples, then write or select an axis that can range from broad to specific.They also choose set size K and diversity strength, from minimal substitutions to richer scene variation.Source: §2 · p. 3
• After inspecting the new set, users can accept it or start another round by changing the axis, strength, set size, or mode.Suggestions are recomputed for the new set.Source: §2 · p. 3
• The user chooses generation or editing, and the corresponding policy produces K prompts or edit instructions for frozen components.Only the prompt-set adapters are trained; the generator, editor, and suggestion VLM stay frozen.Source: §2 · pp. 3–4

3.1 TRAINING DATA

• Training data starts from filtered, clustered HPDv2 prompts and adds VLM-elicited axes grounded in a pilot study, yielding 19,765 training prompts and 76,089 prompt-axis pairs.The data also includes a held-out validation set of 200 prompts.Source: §3.1 · p. 4

3.2 POLICY TRAINING

• The set-level reward combines normalized TPIPS-based diversity with penalties for alignment loss, prompt length, and human preference.Alignment and preference thresholds are calibrated against each prompt’s mean score on i.i.d. samples; the prompt-length budget allows at most w added words.Source: §3.2 · pp. 4–5
• The editing policy writes one instruction per source image and adds a drift penalty, allowing on-axis edits while discouraging replacement of the whole image.Source: §3.2 · p. 5
• GRPO compares groups of full prompt sets, with all sets sharing an initial latent so reward differences reflect policy text rather than sampling noise.Source: §3.2 · p. 5

4.1 SETUP

• Experiments span open-weight, distilled, and proprietary image models, with Qwen3.5 prompt-set policies and evaluation on held-out prompts, real-use interactions, and COCO-250.Policies use LoRA adapters while the T2I model remains unchanged.Source: §4.1 · pp. 5–6

4.2 BASELINES

• Comparisons include untrained and trained T-DIG alongside i.i.d. sampling, PAG, CADS, and Scaling Group Inference, which do not accept a user-specified axis.Scaling Group Inference progressively prunes 128 candidates and costs 3.3–21.9× i.i.d. sampling’s denoiser cost.Source: §4.2 · p. 6

4.3 METRICS

• Normalized TPIPS-Vendi conditioned on the requested axis is the primary diversity metric; alignment, axis-agnostic spread, preference, and aesthetics are also measured.TPIPSv measures variation along a text condition, while DINOv and DreamSimv measure axis-agnostic spread.Source: §4.3 · p. 6

4.4 HUMAN EVALUATION

• Human evaluation uses 405 participant-written prompt–axis pairs and pairwise ratings of diversity and alignment for image sets from trained and untrained T-DIG and baselines.Separate raters score variation along the requested axis and prompt match; majority-N/A pairs are dropped, and results are reported as diversity and diversity-plus-alignment win rates.Source: §4.4 · pp. 6–7

RQ1: Text-conditioned diversity

• T-DIG training raises TPIPSv by +0.14 to +0.26 versus i.i.d. sampling, with at most 0.02 VQAScore lost per +0.1 TPIPSv; baselines lack axis conditioning.The interactive test-set results are reported for K=4 and a 9B policy.Source: §5 · p. 7, Table 1 · p. 8

RQ2: Variation across models

• At α=1, all four models reach TPIPSv 0.35–0.40; gains are largest for Qwen-Image and FLUX.2 [klein], while continued training can trigger reward hacking.T-DIG loses 0.03 and 0.05 VQAScore on Qwen-Image and FLUX.2 [klein], respectively; its alignment difference is not statistically significant on SD3.5 L and M.Source: §5 · p. 7

RQ3: Overall diversity

• On Qwen-Image COCO-250, T-DIG gains +0.21 to +0.39 DINOv while losing 0.01 to 0.02 VQAScore; across four models, it loses 0.006 to 0.010 VQAScore per +0.1 DINOv.Within a 0.02 VQAScore budget, it gains +0.39 DINOv, versus at most +0.11 for any baseline.Source: §5 · pp. 7–8

RQ4: Policy size

• All policy sizes reach nearly the same diversity; 9B matches or beats 4B on VQAScore for every model, while 35B-A3B adds no consistent gain.Training removes the untrained 4B policy’s 1.7% malformed-completion rate, making the 4B policy a lower-latency option with little quality cost.Source: §5 · pp. 8–9

RQ5: Human ratings

• In Stage 1, 89% found asking for variations more useful than i.i.d. outputs; in Stage 2, T-DIG was more varied and at least as aligned in 57–83% of i.i.d., 61–81% of CADS, and 43–76% of Scaling Group Inference comparisons.In Stage 1, mean diversity and prompt-alignment ratings were 3.95/5 and 4.09/5, respectively; 96% enjoyed using the tool.Source: §5 · p. 9

RQ6: Reward ablation

• The full reward keeps TPIPSv at 0.454, near 0.458 with the alignment penalty alone, while diversity-only training raises TPIPSv to 0.617 but drops VQAScore to 0.228.The alignment and length penalties alone still permit reward hacking, including writing the base prompt as an image caption.Source: §5 · p. 9, Table 2 · p. 9

RQ7: Latency

• Expanding a prompt into four takes a median 0.40–0.59 seconds across policy sizes; the 9B policy adds 6–12% to rendering with SD3.5 and Qwen-Image.For 4-step FLUX.2 [klein], the 4B policy adds about 27% overhead.Source: §5 · p. 9

RQ8: Transfer to proprietary models

• Policies trained on open-weight models transfer to GPT Image 2 and Gemini 3.1 Flash Image, with Qwen-Image raising TPIPSv over i.i.d. by +0.27 and +0.20, respectively.All three policies beat proprietary LLM prompt expansion on TPIPSv, with alignment-score drops also reported.Source: §5 · p. 9

Diversity in diffusion models

• Quality, conditioning, and distillation techniques can reduce diffusion-output diversity, especially at large guidance scales.The passage cites quality-improvement methods (Ho & Salimans, 2022; Dhariwal & Nichol, 2021), conditioning methods (Zhang et al., 2023; Voynov et al., 2023; Parmar et al., 2024; Ye et al., 2023; Parmar et al., 2025), and distillation methods (Sauer et al., 2024; Yin et al., 2024b;a; Kang et al., 2024).Source: §6 · p. 10

T2I diversity methods

• Sampler interventions, candidate-pool pruning, fine-tuning, and prompt adaptation each use a fixed notion of diversity rather than a free-form user request.Their constraints differ: sampler interventions need latent or conditioning access and work less well with few-step distilled samplers; pruning costs grow with pool size and is limited to model-produced candidates; fine-tuning requires weights and gradients.Source: §6 · p. 10

Text-conditioned diversity

• Prior diversity work often uses scalar or fixed-attribute measures, and no prior method is trained to vary a set along a user-written condition or evaluated on human-written prompts and axes.TPIPS (Wang et al., 2026) evaluates free-form text-conditioned diversity, while this passage distinguishes that evaluation task from generating it.Source: §6 · p. 10

7 CONCLUSION

• T-DIG makes image sets vary substantially more along requested axes on every tested T2I model, with little loss of alignment.The authors report that existing methods vary sets generically and often at a cost to alignment and image quality.Source: §7 · p. 10
• T-DIG transfers to proprietary models without retraining, and human raters consistently judge its sets more varied than every baseline.Source: §7 · p. 10
• The authors construct and release training and validation data from human-written prompts and axes, plus a test set of 400 real user requests.Source: §7 · p. 10

8 LIMITATIONS

• T-DIG cannot consistently vary camera angles and framing because prompt-level control alone cannot fix these limitations of the base generator.Jointly tuning the generator with the policy is proposed as a possible next step.Source: §8 · pp. 10–11
• Participants found diversity strength poorly calibrated: low settings felt too subtle, while high settings felt too sensitive.The strength setting linearly scales the LoRA weights; the authors suggest a perceptually calibrated mapping to α.Source: §8 · p. 11
• Because training optimizes TPIPS, trained adapters may inherit that metric’s biases.The authors cite this as one reason for collecting a test set through human interactions and human ratings of generated outputs.Source: §8 · p. 11

ETHICS STATEMENT

• T-DIG inherits the underlying T2I model’s safety filters and adds none of its own, although users may request sensitive-attribute variation that could produce stereotyped depictions.Human studies used informed consent and fair compensation, and released data contains no personally identifying information.Source: ETHICS STATEMENT · p. 11

• GPT-5.6 Luna filtered HPDv2 prompts, Gemini 3.8 Flash generated training and validation axes, and fine-tuned Qwen3.5 models formed the policies.LLM assistants also helped edit the paper, prepare figures, and search for related work; AI coding agents helped implement experiments designed by the authors.Source: AI USAGE STATEMENT · p. 11

A.1 TRAINING DATA CONSTRUCTION DETAILS

• The final pool contains 23,020 prompts, one centroid-nearest prompt from each cluster of 82,861 retained prompts, clustered at cosine-distance threshold τ = 0.35.The authors selected τ = 0.35 after inspecting merged pairs and observing broad topic coverage.Source: §A.1 · p. 15
• Figures 4–8 show prompts for HPDv2 filtering, image-conditioned axis elicitation, and GRPO training of generation and editing policies.Figure 3 compares i.i.d. sampling with T-DIG (α=0.5) on interactive-test requests using participants’ prompts and axes verbatim.Source: Figure 3 · p. 16, Figure 4 · p. 17, Figure 5 · p. 18

A.2 PARTICIPANT FEEDBACK

• Camera-angle variation was a frequent failure, and participants also wanted to vary one selected image or preserve and change only image regions.Requests included varying one chosen picture and retaining a specific person or object position while editing other content.Source: §A.2 · pp. 23–24
• Stronger variety could change unrequested base-prompt details, while variation along an axis could repeat values instead of covering distinct options.Feedback included lost core instructions and repeated ethnicity choices despite variation along the requested axis.Source: §A.2 · p. 23
• Participants reported that style changes could alter content instead, while pose, action, and implied context were sometimes insufficiently varied.Feedback included requests for more artistic-style variation, dynamic poses, and differences implied by the prompt but not explicitly named in the axis.Source: §A.2 · p. 23
• Counts, rare concepts, and specific products could fail in the renderer’s i.i.d. samples, including requests for exact clock numerals or dancer counts.The feedback recommends identifying when the renderer cannot express a request rather than varying around it.Source: §A.2 · p. 23
• Participants said artifacts accumulated over repeated image edits and described the variety-strength slider as difficult to control.One participant characterized its settings as either very subtle or outrageous.Source: §A.2 · pp. 23–24