GenEval 2: Addressing Benchmark Drift in Text-to-Image Evaluation

1 Introduction

• GenEval scores for recent models diverge from human judgment by as much as 17.7%, and human evaluation finds the benchmark saturated, with Gemini 2.5 Flash Image scoring 96.7%.The study collected over 23,000 annotations on outputs from eight popular T2I models released over three years; Figure 2 shows net deviation increasing over time.Source: §1 · p. 2, Figure 2 · p. 3
• GenEval 2 expands concept coverage and compositionality while retaining GenEval’s basic-capability focus and templated design.Human annotations identify struggles with spatial relations, transitive verb relations, and counting, and show performance tending to drop as prompts become more compositional.Source: §1 · p. 2
• The top-performing model achieves only 35.8% prompt-level accuracy on compositional prompts, despite performing well on some individual atoms.This indicates substantial room for improvement on GenEval 2’s compositional prompts.Source: §1 · p. 2
• On GenEval 2 prompts, Soft-TIFA achieves 94.5% AUROC with Qwen3-VL-8B, versus 92.4% for VQAScore and 91.6% for TIFA.It combines judgments for visual primitives to estimate atom- and prompt-level performance; its alignment remains consistent over time while VQAScore-based estimates tend to diverge from human judgments after a VQA model’s release.Source: §1 · p. 2
• The authors argue that T2I evaluations need continual auditing and updating to maintain validity as benchmarks drift.They also connect compositionality with controllability: correctly distinguishing prompts that swap the colors of two animals entails greater user control.Source: §1 · pp. 2–3

2 Related Work

• T2I benchmarks cover basic capabilities such as counts, colors, positions, and attribution; scene-level verbs and information; and advanced world knowledge and reasoning.Examples include GenEval [Ghosh et al., 2023], GenAI-Bench [Li et al., 2024], and WISE [Niu et al., 2025].Source: §2 · p. 3
• Evaluation has moved from specialized detectors and image-text matching toward VQA-based methods, which answer questions about generated images.TIFA [Hu et al., 2023] decomposes prompts into questions, while VQAScore [Lin et al., 2024] uses a single question containing the prompt.Source: §2 · p. 3
• The authors call for considering benchmark drift and present a challenging benchmark for frontier T2I models alongside an evaluation method less susceptible to drift.Source: §2 · p. 3

3.1 Preliminary: GenEval Evaluation Method

• GenEval evaluates 553 prompts across six basic-capability categories using category-specific model-based checks, but its position calculations can mislabel valid overlapping objects.The method uses MaskFormer for objects, CLIP for color, and bounding-box coordinates for relative position; background masking can also affect color estimates.Source: §3.1 · p. 4

3.2 A New Human-Alignment Study of GenEval

• The human-alignment study annotated 23,226 images or prompt variants across eight models, with three annotators per data point and majority-vote labels.The models span releases from 2022–25; rewritten prompts were evaluated for six models, while SD2.1 and SDXL were not trained with rewriting.Source: §3.2 · pp. 4–5

3.3 Findings from the Human Alignment Study of GenEval

• GenEval scores are 11.0% lower than human scores on average for today’s state-of-the-art models, reaching a 17.7% gap for Gemini 2.5 Flash Image.The gap grew over time: it was within ±3% for SD2.1 and SDXL, then reached 17.7% for Gemini 2.5 Flash Image.Source: §3.3 · p. 5
• Across six models trained with rewriting techniques, rewritten prompts reduced human alignment by 2.5% on average for four models, while increasing it by 3.3% on average for Qwen-Image and Gemini 2.5 Flash Image.Source: §3.3 · p. 5
• All four state-of-the-art models score at least 94.8% on GenEval with rewriting, indicating that the benchmark is saturated today and has been for over a year.This conclusion is based on human scores for models released in 2024–25.Source: §3.3 · p. 5

4.1 Benchmark Curation

• GenEval 2 covers 40 objects, 18 attributes, 9 relations, and 6 counts, using templated prompts with one to three objects and optional components.The object set includes animate and inanimate objects; relations include 3D spatial prepositions and transitive verbs.Source: §4.1 · pp. 5–6
• Its 800 prompts span atomicities from 3 to 10, with 100 prompts at each level; atomicity counts object, attribute, and relation atoms.Prompts are also rewritten with GPT-4o, though CLIP’s 77-token input limit truncates some rewritten prompts.Source: §4.1 · p. 6
• Human annotations cover half the benchmark across eight models and rewriting conditions, with atom-level checklists and one annotator per image after a 91% unanimous pilot.The prompt-level score is binary: every atom must be labeled correct for the prompt to count as correct.Source: §4.1 · p. 6, §4.2 · p. 7

4.2 Human-Judged Scores on GenEval 2

• 85.3% atom-level human-judged performance contrasts with only 35.8% prompt-level performance for the highest-performing model, leaving room to improve on compositional prompts.A prompt counts as correct only when every component atom is correct.Source: §4.2 · p. 7, Table 2 · p. 7
• Rewriting improves atom-level and prompt-level performance across all models, with the increase for Gemini 2.5 Flash Image appearing only in the second decimal.The table caption notes that CLIP-based models truncate rewritten prompts at 77 tokens.Source: §4.2 · p. 7, Table 2 · p. 7

4.3 Analyses of Capability and Compositionality

• Models handle objects and attributes better than counting, spatial relations, and transitive verbs, while performance drops sharply as prompt complexity increases.At the highest complexity levels, average performance is close to zero; Qwen-Image, for example, scores 99% on specific objects but 60% on relative positions.Source: Figure 4 · p. 7, §4.3 · p. 8

5.1 Preliminaries: VQAScore and TIFA

• VQAScore asks a VQA model whether an image shows the prompt and scores its “Yes” probability, whereas TIFA generates prompt-based question-answer pairs and scores predicted answers.Soft-TIFA draws on both methods’ soft scoring and question-based evaluation ideas.Source: §5.1 · p. 8

5.2 Soft-TIFA

• Soft-TIFA generates one template-based question per prompt atom and assigns soft VQA scores, using arithmetic or geometric averaging to estimate atom-level or prompt-level correctness, respectively.The geometric mean penalizes low atom-level scores at the prompt level.Source: §5.2 · p. 9

5.3 Human Alignment of Soft-TIFA on GenEval 2

• Soft-TIFAGM outperforms existing methods on original, rewritten, and combined GenEval 2 prompts, while Soft-TIFAAM matches or outperforms them on rewritten and combined prompts.The authors attribute Soft-TIFA’s alignment to its structure for capturing varying prompt compositionality.Source: §5.3 · p. 9

5.4 Comparison of T2I Models

• Soft-TIFA largely retains the human-scored ranking of T2I models, while the four state-of-the-art models score well at the atom level but have room to improve at the prompt level.The comparison uses Soft-TIFAAM for atom-level correctness and Soft-TIFAGM for prompt-level correctness.Source: §5.4 · p. 10, Table 4 · p. 10

5.5 Evaluation Methods vs. Benchmark Drift

• VQAScore human alignment declines on T2I models released after its underlying VQA model, while Soft-TIFAGM shows a smaller decline but may still drift over longer periods.The authors suggest per-atom questions may make VQA judgments less sensitive to image-distribution shift.Source: §5.5 · p. 11
• Moving from specialized evaluation models to VQA-based methods alone is insufficient to address benchmark drift, so evaluation components need updating as T2I outputs change.The authors’ comparison found declining human alignment for VQAScore under output-distribution shifts.Source: §5.5 · p. 11

6 Conclusion

• GenEval’s results on recent models diverge from human judgment by as much as 17.7% absolute error, and human evaluation indicates that the benchmark is saturated.The study examines output-distribution shifts over the past three years.Source: §6 · p. 11
• GenEval 2 evaluates basic capabilities on increasingly compositional prompts, supports targeted analyses, and remains challenging for state-of-the-art T2I models.Human annotations show those models still have significant room for improvement.Source: §6 · p. 11
• Soft-TIFA estimates atom-level and prompt-level performance and has higher human alignment than VQAScore and TIFA, with greater robustness to benchmark drift.The authors stress the need to continually audit and update T2I evaluation benchmarks.Source: §6 · p. 11

A Example of a Failure Case of the GenEval Detector

• GenEval’s COCO-trained detector can break down as T2I output distributions shift; in one example, it labels a correctly placed toothbrush as a bird.The example attributes the error as likely related to the detector’s learned associations between toothbrushes and bathrooms and birds and skies.Source: A Example of a Failure Case of the… · p. 15, Figure 6 · p. 15

B.1 Full Vocabulary of GenEval 2

• GenEval 2 prompts are randomly sampled from a vocabulary spanning animate and inanimate objects, attributes, prepositions, transitive verb relations, and counts.Objects sourced from COCO are identified in the vocabulary.Source: §B.1 · p. 15

B.2 GPT-4o Rewriting Prompt

• For longer, more detailed GenEval 2 prompts aligned with recent text-to-image training and evaluation methods, authors use Figure 7’s prompt and BAGEL repository8 examples; 550 of 553 rewrites fit the suggested 110-word maximum.Source: §B.2 · p. 16

C User Study Details

• The appendix provides annotator instructions and user interfaces for the two studies conducted in the work.Source: C User Study Details · p. 16

• Annotators see an image and answer separate yes/no questions about prompt alignment and whether its objects are well-formed.Source: §C.1 · pp. 16–17
• Annotators should judge alignment and quality independently; unrealistic scenes may still be high quality, while malformed objects are not.A partially visible object counts as identifiable when it can be clearly and unambiguously recognized.Source: §C.1 · p. 17
• For position prompts, annotators judge from the viewer’s perspective; the instructions distinguish an object being above another in 3D space from appearing above it in the image plane.An unusual perspective can still satisfy the requested relative position.Source: §C.1 · pp. 17–18

C.2 GenEval 2 User Study

• Annotators mark alignment for each atom in the prompt, yielding atom-level correctness annotations for each model output.The GenEval study instead used a single yes/no alignment question.Source: §C.2 · p. 19
• The study task also asks annotators whether the image is well-formed, using a yes/no image-quality question alongside alignment assessment.The first question is answered part-by-part; the second is yes/no.Source: §C.2 · p. 19
• Figure 11 presents annotator instructions, and Figure 12 shows the GenEval 2 user-study interface.Source: Figure 11 · p. 19, Figure 12 · p. 19

D Per-Model Analyses of Human-Judged Scores on GenEval 2

• Gemini’s counting accuracy is 69.2%, while Qwen-Image scores 61.1% on both spatial and transitive-verb relations.Table 5 reports per-model performance across skills, enabling targeted diagnosis of model flaws.Source: D Per-Model Analyses of Human-Judged… · p. 20, Table 5 · p. 20
• All models perform better on less complex prompts; Flux.1-dev reaches 25% accuracy at atomicity 6 and 0% at every higher atomicity.Table 6 presents per-model results by prompt atomicity, highlighting room for improvement at higher compositionality.Source: D Per-Model Analyses of Human-Judged… · p. 20, Table 6 · p. 20

E Illustrative Example of Soft-TIFA

• Soft-TIFA combines soft scores with per-atom questions and uses GenEval 2’s templates to cover the full prompt.The example distinguishes Soft-TIFA_AM, an estimate of atom-level performance, from Soft-TIFA_GM, an estimate of prompt-level performance.Source: E Illustrative Example of Soft-TIFA · p. 21, Figure 13 · p. 21

F Evaluation Methods with Different Underlying VQA Models

• The comparison evaluates four methods across prompt sets and underlying VQA models using AUROC.Source: F Evaluation Methods with Different… · p. 21, Table 7 · p. 21
• Soft-TIFA_GM has higher human alignment than the other methods with every tested Qwen model across GenEval 2 prompt types.Table 7 compares AUROC for VQAScore, TIFA, Soft-TIFA_AM, and Soft-TIFA_GM across original, rewritten, and combined prompt sets.Source: F Evaluation Methods with Different… · p. 21, Table 7 · p. 21
• With GPT-4o as the VQA model, VQAScore outperforms Soft-TIFA_GM, but its AUROC is significantly lower than Soft-TIFA_GM with Qwen models.Source: F Evaluation Methods with Different… · p. 21, Table 7 · p. 21

G Per-Skill and Per-Compositionality Analyses of Soft-TIFA

• Soft-TIFA_AM estimates performance on atom-level skills, while Soft-TIFA_GM estimates performance on prompt-level compositionality.The analyses cover four state-of-the-art models: Flux.1-dev, Bagel, Qwen-Image, and Gemini 2.5 Flash.Source: G Per-Skill and Per-Compositionality… · p. 22
• Because Soft-TIFA scores are soft rather than binary, high scores are slightly lower and low scores slightly higher; the analysis focuses on trends.Source: G Per-Skill and Per-Compositionality… · p. 22
• Soft-TIFA trends largely mirror human-judged results, but for verb skill its VQA model scores 31.4 versus 65.0 for human judgment.Figure 14 presents atom-level skill and prompt-level compositionality analyses; its trends reflect the human-judged scores in Figure 4 except for verb.Source: G Per-Skill and Per-Compositionality… · p. 22, Figure 14 · p. 22