Source-linked AI summary
MineTheGap: Automatic Mining of Biases in Text-to-Image Models
Noa Cohen, Nurit Spingarn-Eliezer, Inbar Huberman-Spiegelglas, Tomer Michaeli
TL;DR
TTI models can interpret ambiguous prompts in biased and semantically narrow ways, motivating automatic discovery beyond manually selected prompts or bias axes. MINETHEGAP uses an LLM-driven genetic algorithm and a distributional bias score to search prompts for such outputs. The method identifies prompts judged more biased than random prompts and aligns with known-bias rankings, while relying on approximations from LLM text and CLIP features.
Problem
TTI models may overlook valid semantic interpretations and reflect or amplify biases, while existing methods do not provide a unified bias evaluation across interacting axes.
Source: §1 · pp. 1–2Method
MINETHEGAP uses genetic optimization over prompts, guided by a score comparing generated-image distributions with distributions from diverse LLM-generated textual variations.
Source: §1 · pp. 1–2, §3 · p. 3Results
Mined prompts were consistently judged more biased than random prompts, and the bias score correlated with known-bias rankings at ρ = 0.72 using CLIP.
Source: §4.2 · p. 7, §5 · p. 9Takeaways & Limitations
Automatic prompt mining can reveal both known and previously unseen open-set biases in TTI models without relying on predefined captions, concepts, or bias categories.
Source: §1 · p. 2The method relies on an LLM to approximate the target distribution and on CLIP similarity, restricting analysis to biases represented by those models’ sensitivities.
Source: §6 · p. 10
Abstract
from arXiv · showhide
Text-to-Image (TTI) models generate images based on text prompts, which often leave certain aspects of the desired image ambiguous. When faced with these ambiguities, TTI models have been shown to exhibit biases in their interpretations. These biases can have societal impacts, e.g., when showing only a certain race for a stated occupation. They can also affect user experience when creating redundancy within a set of generated images instead of spanning diverse possibilities. Here, we introduce MineTheGap - a method for automatically mining prompts that cause a TTI model to generate biased outputs. Our method goes beyond merely detecting bias for a given prompt. Rather, it leverages a genetic algorithm to iteratively refine a pool of prompts, seeking for those that expose biases. This optimization process is driven by a novel bias score, which ranks biases according to their severity, as we validate on a dataset with known biases. For a given prompt, this score is obtained by comparing the distribution of generated images to the distribution of LLM-generated texts that constitute variations on the prompt. Code and examples are available on the project's webpage.
1 INTRODUCTION
MINETHEGAP addresses the gap between human-expected and model-generated semantic diversity by automatically mining prompts that elicit biased TTI outputs. It combines a bias measure based on plausible textual variations with genetic optimization over prompts.
1 INTRODUCTION
- The method targets a gap in semantic diversity: TTI models can produce realistic, prompt-adherent images while overlooking alternative valid interpretations.Such gaps can reflect or amplify sociodemographic biases learned from training data.Source: §1 · p. 1
- MINETHEGAP automatically searches valid prompts to find those that cause TTI models to produce biased outputs.An LLM initializes candidate prompts, and successive iterations retain and refine the most biased ones.Source: §1 · p. 1
- Its bias measure compares generated-image distributions with distributions of plausible textual variations to capture deviations across interacting semantic axes.Because the intended distribution is unobservable, diverse LLM-generated variations approximate it.Source: §1 · p. 2
- The work contributes both automatic discovery of known and previously unseen open-set biases and a bias measure that avoids dependence on predefined attributes or categories.The measure compares image distributions with plausible textual variations rather than fixed bias axes.Source: §1 · p. 2
2 RELATED WORK
Prior TTI-bias research includes predefined-axis detection, open-set methods, output-diversity analyses, and prompt optimization. MINETHEGAP differs by searching the unconstrained prompt space for prompts that induce biased outputs.
2 RELATED WORK
- Existing TTI-bias methods often target predefined categories such as race and gender, while open-set methods seek biases without fixing them in advance.Related work also examines output diversity and mitigation of detected biases.Source: §2 · pp. 2–3
- Figure 2 illustrates that bias can combine broad gender skew with model-specific tendencies such as grayscale imagery in SD 2.1 and masks in FLUX.The comparison uses generated images for “a photo of a nurse” against variations spanning gender, race, style, and surroundings.Source: Figure 2 · p. 3
- D’Inc`a et al. (2024) evaluate open-set biases on a fixed set of real captions, restricting findings to that caption set.MINETHEGAP instead identifies bias-inducing prompts from the unconstrained textual prompt space.Source: §2 · p. 3
- Genetic algorithms have optimized prompts for image quality and text-image consistency, but prior approaches generally refine prompts within a constrained search space.MINETHEGAP searches the entire prompt space to uncover prompts that inherently induce biased outputs.Source: §2 · p. 3
3 AUTOMATIC MINING
MINETHEGAP performs gradient-free evolutionary search over the discrete space of valid TTI prompts. It iteratively selects high-scoring candidates, mutates them, injects random prompts, and retains the best prompts found.
3 AUTOMATIC MINING
- The framework optimizes any objective over the high-dimensional, nondifferentiable prompt space through iterative population selection and mutation.Bias mining uses the paper’s bias objective, but the optimization framework is objective-independent.Source: §3 · pp. 3–4
- Each iteration ranks the population, carries forward the s lowest-loss prompts, generates mutations for them, and retains the best prompts across all iterations.Lower objective values indicate stronger bias in the bias-mining setting.Source: §3 · pp. 3–4, Figure 3 · p. 4
- An LLM generates an initial population of b diverse prompts intended to broadly cover the valid prompt space.The candidates are returned in one query, parsed, and used as the starting population.Source: §3 · p. 3
- Mutation introduces visually distinct alternatives through substitutions, omissions, or substantial changes while preserving a connection to each selected prompt.For example, “a doctor” may mutate into “a nurse” or “a dentist.”Source: §3 · p. 4
- Random prompt injection preserves exploration by preventing population homogeneity and uses r = b − s × m candidates to maintain the fixed population size.The injected candidates are independently generated by the LLM.Source: §3 · p. 4
4 RANKING BIAS
MINETHEGAP ranks prompt bias by comparing generated-image diversity with diverse LLM-generated textual interpretations, then validates the score against known occupational biases and controllable diversity changes.
4. RANKING BIAS
- The resulting normalized average provides an interpretable ranking whose missing concepts and least-aligned images indicate the source of a prompt’s bias.The method is designed to rank prompts without specifying the bias type in advance.Source: §4 · p. 4, §4.1 · p. 6
4.1 MEASURING BIAS THROUGH LEAST-ALIGNED ELEMENTS
- The score compares generated images with LLM-generated prompt variations in a shared latent space to identify missing concepts and poorly aligned outputs.Lower-percentile maxima quantify both missed textual concepts and generated images lacking strong textual correspondences.Source: §4 · p. 5, §4.1 · pp. 5–6
4.2 VALIDATING OUR BIAS SCORE
- For occupational prompts, the metric’s visual ranking agrees with BLS-based gender bias examples, including air conditioning installer as most biased and teller as least biased.The evaluation uses profession prompts and gender-and-race textual variations.Source: Figure 5 · p. 6, §4.2 · p. 6
- ρ = 0.72 with CLIP versus Luccioni et al. (2024)’s ρ = 0.68, while SigLIP and SigLIP-2 achieve ρ = 0.63 and ρ = 0.62.All variants positively correlate with occupational ground truth, and CLIP is adopted as the shared latent space.Source: §4.2 · p. 7
- The score tracks expected diversity: higher classifier-free guidance produces more biased outputs across three Stable Diffusion models.The experiment uses the same seeds and LLM-generated variations across CFG settings; complementary CADS results show increased diversity yields less biased scores.Source: Figure 6 · p. 6, §4.2 · p. 7
5 EXPERIMENTS
MINETHEGAP uses an LLM-guided genetic search to mine biased prompts across multiple TTI models. The mined prompts consistently induce repetitive or stereotyped outputs, align with human judgments, and reveal model-specific and cross-model bias patterns.
5 EXPERIMENTS
- Mined prompts consistently attain lower bias losses and lower variance than random prompts, indicating stronger and more stable bias under the genetic search.The comparison also includes captions from COCO.Source: §5 · p. 7
- Mined prompts exposed repetitive semantics, including male-only scientists, photorealistic-only images, grass-bound dogs, and violet-focused scenes despite broader textual variations.The variations demonstrate that the models can generate additional interpretations when prompted directly.Source: §5 · p. 7, Figure 7 · p. 8
- Adjusting the LLM meta-prompts restricted mining toward specified bias types such as gender and age.The modified variations preserve the original prompt’s meaning while allowing targeted demographic differences.Source: §5 · p. 8, Figure 7 · p. 8
- Each model showed its strongest bias on its own mined prompts, while SD 1.4 ranked least biased and FLUX.1 Schnell most biased across the four models.Bias remained relatively low on other models’ mined prompts, suggesting shared bias types across models.Source: §5 · p. 8, Figure 9 · p. 9
- Mined prompts were consistently perceived as more biased, and score differences correlated with human judgments at Pearson r = 0.71.Only four of 40 questions had a majority vote favoring the mined prompt as less biased.Source: §5 · p. 9, Figure 10 · p. 9, Figure 11 · p. 9
- Setting-related bias categories were most frequent, followed by nature-related categories, while style biases were less frequent but could be targeted directly.Categories are assigned from missed visual concepts absent from the original prompts.Source: §5 · pp. 9–10, Figure 12 · p. 10
- Compared with OpenBias, MINETHEGAP surfaced location and child-presence biases that closed-set person-focused queries could miss, achieving ρ = 0.72 versus OpenBias’s ρ = 0.64.OpenBias’s gender, activity, and age questions can be irrelevant when no person appears, while food type may already be specified.Source: §5 · p. 10, Figure 13 · p. 10
6 CONCLUSION
MINETHEGAP automatically discovers prompts that lead TTI models to produce biased outputs, while highlighting limitations in how its target distribution and visual-textual gap are estimated.
6 CONCLUSION
- MINETHEGAP automatically finds prompts that elicit systematically biased outputs, targeting overlooked but valid visual interpretations.The method uses LLM-driven prompt exploration and gradient-free optimization.Source: §6 · p. 10
- The approach relies on an LLM that may itself encode bias when approximating the target distribution.The authors also note that CLIP restricts analysis to features it can detect.Source: §6 · p. 10
A ANALYSIS OF THE OPTIMIZATION PROCESS
The optimization converges quickly on a synthetic color-matching task and consistently finds more biased prompts than random prompt generation.
A ANALYSIS OF THE OPTIMIZATION PROCESS
- Within three to four iterations, optimization finds prompts producing images dominated by a target red, blue, or green color.The synthetic objective minimizes mean squared error between generated and target-color images.Source: A ANALYSIS OF THE OPTIMIZATION PROCESS · p. 14, Figure S1 · p. 14
- Mined prompts consistently achieve lower bias scores than COCO captions and randomly initialized prompts across models.Lower scores indicate stronger measured biases, while reduced variance suggests convergence toward a stable set of high-bias prompts.Source: A ANALYSIS OF THE OPTIMIZATION PROCESS · p. 14
- Across 25 iterations, mining achieves lower losses than repeatedly generating disjoint random prompt sets with an LLM.The comparison averages the best loss over 10 runs for each model.Source: A ANALYSIS OF THE OPTIMIZATION PROCESS · p. 14
A.1 ABLATION STUDY ON OPTIMIZATION HYPER-PARAMETERS
The ablations examine population composition, population size, and score sensitivity, while the user study and occupational benchmark assess alignment with human judgments and known societal patterns.
A.1 ABLATION STUDY ON OPTIMIZATION HYPER-PARAMETERS
- The five prompt-composition configurations perform broadly similarly, so the paper uses s = 5, m = 2, r = 5 as a balance between exploitation and exploration.The batch size is fixed at b = 15, and FLUX.1 Schnell performs best without random candidate injection.Source: §A.1 · p. 14, Table S1 · p. 15
- Larger populations identify lower-loss, more biased prompts earlier, but increase the total number of NFEs.The tested configurations use population sizes b = 9, 15, and 21; the standard setting is b = 15.Source: §A.1 · pp. 15–16, Figure S4 · p. 16
- The bias-score correlation with human judgments remains positive across α values and peaks at the 25th percentile used in the paper.Lower α values mechanically reduce loss, so loss alone is not informative for selecting α.Source: §A.1 · p. 16, Figure S5 · p. 16
A.2 USER STUDY
- Mined prompts are consistently perceived as more biased than random prompts in a study with 1,200 responses across four TTI models.Participants compared 15-image sets from mined and random prompts in 40 questions.Source: §A.2 · p. 17, Figure S6 · p. 17
B.1 SOCIETAL BIAS RANKING
- Against real-world occupational statistics, our bias score shows a positive Spearman’s rank correlation with ground truth and slightly stronger alignment than Luccioni et al. (2024)’s ranking.Source: §B.1 · p. 18
B.2 THE EFFECT OF CADS ON MEASURED BIAS
CADS increases measured diversity and reduces measured bias most clearly for Stable Diffusion 1.4, while its effect is more limited for Stable Diffusion 2.1.
B.2 THE EFFECT OF CADS ON MEASURED BIAS
- CADS produces higher, less biased average scores for Stable Diffusion 1.4, supporting the validity of the bias metric.The trend is less prominent for Stable Diffusion 2.1, where CADS induces smaller semantic modifications.Source: §B.2 · p. 19, Figure S7 · p. 19
- Qualitative examples show greater output variability with CADS for Stable Diffusion 1.4 than for Stable Diffusion 2.1.The comparison uses identical seeds and 15 images per prompt with and without CADS.Source: §B.2 · p. 19, Figure S8 · p. 20
C EXPERIMENTAL DETAILS
The supplementary experiments specify LLM prompts for candidate generation, mutation, and variation, then evaluate missed visual concepts and semantic bias categories. These procedures compare generated images with plausible prompt interpretations to expose absent concepts.
C.1 META-PROMPTS
- Meta-prompts generate short, diverse candidates and mutations, while variation prompts preserve meaning and explore unspecified subject, style, and setting interpretations.The prompts also require strict list formatting and discourage vague adjectives, punctuation, and minor synonym substitutions.Source: §C.1 · p. 21
C.2 EXAMPLES FOR PROMPT MUTATIONS
- Mutation examples replace subjects or settings to produce visually distinct prompts, such as doctors becoming nurses or dentists and landmarks becoming towers or lighthouses.These examples illustrate exploration through meaningful modifications rather than simple rephrasings.Source: §C.2 · p. 22
C.3 EXAMPLES FOR PROMPT VARIATIONS
- Variation sets approximate human-expected diversity by instantiating plausible meanings, including workplace, gender, activity, setting, and appearance changes for “A photo of a doctor”.The examples contain 15 variations spanning offices, hospitals, patients, male and female doctors, instruments, and clothing.Source: §C.3 · p. 22
- For “A photo of a doctor”, FLUX.1 Schnell misses the female-doctor concept, whereas the other three models each miss different concepts such as patient interaction or emergency-room settings.The method identifies concepts absent from generated images relative to the textual variation set.Source: §C.3 · p. 23, Figure S9 · p. 23
C.4 EVALUATING CATEGORIES FOR MINED PROMPTS
- Category analysis assigns prompts to semantic categories when missed concepts contain category terms absent from the original prompt, allowing one prompt to receive multiple categories.The categories include architecture, spatial relationships, atmosphere and weather, textures and materials, and food and drink.Source: §C.4 · p. 24
D ADDITIONAL RESULTS
Additional qualitative results compare native generations with variation-generated interpretations, showing limited diversity and missed concepts across models. The appendix also examines recurring prompt terms and longer-prompt mining.
D ADDITIONAL RESULTS
- Across models, mined-prompt generations often show limited diversity or narrow prompt aspects, while variation generations are more varied and reveal missed visual concepts.The two 15-image rows provide a visual comparison between native generations and plausible interpretations generated by Llama 3.1-8B-Instruct.Source: D ADDITIONAL RESULTS · p. 24
- Word clouds from the 50 most biased prompts per model show recurring nature-related terms such as “field” and “forest”, which are associated with limited visual diversity.Source: D ADDITIONAL RESULTS · p. 24, Figure S15 · p. 29
- Relaxing the prompt-length constraint from eight to 30 words tests whether MINETHEGAP can mine biases in longer prompts, while variation generation adapts to remaining ambiguity.The framework itself is described as independent of prompt length, although short prompts were deliberately targeted for general biases.Source: D ADDITIONAL RESULTS · p. 24, Figure S16 · p. 30
D.1 MINING WITH DIFFERENT LLMS
MINETHEGAP produces similar mined-prompt distributions across different LLM drivers and can target specified sociodemographic, object, food, clothing, and style biases through controlled meta-prompts.
D.1 MINING WITH DIFFERENT LLMS
- Prompts mined with Llama, LLaDA, and Qwen are more similar to one another than to COCO captions, while t-SNE shows COCO occupying subregions of the semantic space.The comparison uses 100 prompts per LLM and CLIP nearest-neighbor cosine similarity.Source: §D.1 · p. 32, Figure S19 · p. 32, Figure S20 · p. 32
D.2 MINING SPECIFIC BIASES
- Controlling variation prompts for gender, race, and age exposes sociodemographic biases, including exclusively male firefighters and Caucasian, light-haired toddlers despite more diverse textual variations.The targeted examples use SD 2.1 for firefighters and SD 1.4 for toddlers.Source: §D.2 · p. 33, Figure S21 · p. 33
- Separate meta-prompt settings target object, food, clothing, and image-style biases by varying subject features, settings, or visual styles during textual variation generation.The resulting examples include croissants with similar viewpoints and women wearing the same coat style.Source: §D.2 · p. 33, Figure S22 · p. 34
D.3 SUPPLEMENTARY RESULTS FOR THE COMPARISON WITH OPENBIAS
The supplementary OpenBias comparison shows that MINETHEGAP exposes biases that candidate-based VQA analysis can overlook. Its richer variations reveal hidden contextual, compositional, and framing biases in COCO and mined prompts.
D.3 SUPPLEMENTARY RESULTS FOR THE COMPARISON WITH OPENBIAS
- OpenBias overlooks some COCO captions marked highly biased by MINETHEGAP because its candidates can be visually uniform or difficult for VQA to assess.MINETHEGAP instead generates diverse variations that expose cues such as signs, face-paint patterns, and contextual attributes.Source: §D.3 · p. 35, Figure S23 · p. 36
- For purple-and-black striped socks, MINETHEGAP reveals a bias toward showing only the socks by adding surrounding actions and settings beyond OpenBias’s color-focused candidate.The variations place the person with a coffee cup, phone, or building.Source: §D.3 · p. 35
- Across 200 MINETHEGAP-mined prompts, OpenBias identifies only 19 unique bias names, and Figure S24 shows additional mined prompts it fails to identify as biased.Source: §D.3 · p. 35, Figure S24 · p. 36
- For “A hedgehog nestled in a handwoven rattan basket”, MINETHEGAP varies setting and basket function to surface visual concepts that OpenBias treats as integral caption elements.Examples place the hedgehog in storage, planter, picnic, and ivy-lined basket contexts.Source: §D.3 · p. 35
- For “A lioness walking in tall grass”, MINETHEGAP generates varied landscapes and movement contexts, whereas OpenBias candidates remain difficult for VQA to answer.Source: §D.3 · p. 35