EvalMuse-40K: A Reliable and Fine-Grained Benchmark with Comprehensive Human Annotations for Text-to-Image Generation Model Evaluation

1. Introduction

• Traditional metrics such as FID and CLIPScores are not well-suited to T2I consistency, while simple VQA scoring cannot handle fine-grained matching well.Element-based VQA methods enable fine-grained evaluation but are limited by VQA-model performance; their relationship to overall human preferences remains unexplored.Source: §1 · p. 1
• EvalMuse-40K contains 4K prompts, 40K image-text pairs, and more than 1M fine-grained human annotations.Its annotations support comprehensive evaluation of automated metrics on image-text alignment-related tasks.Source: §1 · pp. 1–2
• EvalMuse-40K combines balanced prompts and diverse generative models with fine-grained element and structural-problem annotations.The benchmark is intended to support analysis of automated metrics and their correlation with human preferences.Source: §1 · p. 2
• FGA-BLIP2 jointly fine-tunes overall alignment scores and element-level annotations, while PN-VQA uses positive-negative question-answer pairs for fine-grained evaluation.FGA-BLIP2 also uses variance-weighted optimization to account for prompts’ capacity to generate diverse images.Source: §1 · p. 2
• The authors use their methods to analyze representative AIGC models, and describe the results as resources for further research.The listed contributions also report impressive FGA-BLIP2 performance across multiple datasets and improved correlation with human ratings for PN-VQA.Source: §1 · p. 2

Image-Text Alignment Benchmarks

• Earlier benchmarks are often small and skill-limited, while newer prompt benchmarks target skills such as counting, spatial relations, and attribute binding.Some benchmarks provide human alignment annotations, and fine-grained benchmarks such as TIFA and SeeTRUE extract prompt elements and generate questions.Source: §2 · p. 2
• EvalMuse-40K combines 2K real prompts sampled from DiffusionDB [41] with 2K synthetic prompts for specific skills and large-scale overall and fine-grained human annotations.The paper presents this mix as a way to address the difference between many benchmark prompts and those used by real users.Source: §2 · p. 3

Automated Metrics for Image-Text Alignment

• FID, IS, and LPIPS assess image quality rather than image-text alignment, while CLIPScore and BLIP2Score are used as alignment metrics.Human-preference models based on side-by-side comparisons can have difficulty producing accurate alignment scores and fine-grained evaluations.Source: §2 · p. 3
• FGA-BLIP2 predicts overall and fine-grained alignment scores, while PN-VQA’s positive-negative question template supplies VQA models with more accurate prompt context.The paper contrasts this with generated questions that can diverge from the original prompt’s content.Source: §2 · p. 3

3. EvalMuse-40K Benchmark

• EvalMuse-40K construction uses strategies intended to ensure benchmark diversity and reliability across prompt collection, image generation, and data annotation.The section also reports statistical analysis of the resulting data.Source: §3 · p. 3

3.1. Prompts Collection

• The benchmark uses 2K real prompts sampled from DiffusionDB [41] and 2K synthetic prompts generated for six image-text alignment skill categories.Real prompts are classified across subject, logical relation, style, and BERT embedding aspects; MILP sampling aims for approximate category balance. Prompts are split into elements and paired with filtered yes/no questions.Source: §3.1 · p. 4

3.2. Image Generation

• Images for the benchmark are generated from 4K prompts using 20 diffusion-based T2I models, with a randomly sampled model subset per prompt.The selected models span four groups, including basic stable diffusion, advanced open-source, efficient, and proprietary models.Source: §3.2 · p. 4

3.3. Data Annotation

• Annotations cover 5-point overall alignment, element matching, structural issues, splitting confidence, and prompt meaningfulness.Each pair is labeled by three annotators plus quality control; pairs with alignment-score range ≥2 are re-annotated, yielding 3 to 6 annotators and an averaged final score.Source: §3.3 · pp. 4–5

3.4. Data Statistics and Reliability Analysis

• 75% of samples have annotator score differences under 1 point; most fine-grained categories have alignment scores around 50%, while counting, shape, and activity show relatively poor text consistency.The benchmark contains 40K generated images and nearly 1M annotations; alignment scores concentrate more in the middle range.Source: §3.4 · p. 5

4.1. End-to-End Scoring Model: FGA-BLIP2

• FGA-BLIP2 jointly scores overall and element-level alignment, using a token-validity mask to identify which text tokens matter for alignment.Image-text representations yield an overall score from query tokens and element-specific scores from corresponding text positions.Source: §4.1 · pp. 5–6, Figure 3 · p. 6
• Prompt-variance weighting raises the loss for examples with greater alignment-score differences among images generated from the same prompt.The total loss combines overall-score, element-score, and valid-token losses; λ and η are each set to 0.1.Source: §4.1 · p. 6

4.2. Fine-Grained Zero-Shot Alignment: PN-VQA

• PN-VQA evaluates elements with positive and negative yes/no questions, using the MLLM’s probabilities for correct and incorrect answers.The fine-grained score averages the probability of the correct answer with one minus the probability of the incorrect answer; including the original prompt helps assess element-image alignment.Source: §4.2 · p. 6

5.2. Results on Automated Metrics

• FGA-BLIP2 shows higher correlation with human annotations than other methods on EvalMuse-40K, GenAI-Bench, TIFA800, and RichHF alignment scores.The comparison covers methods that output overall alignment scores directly from image-text pairs.Source: Table 2 · p. 7, §5.2 · p. 7
• FGA-BLIP2 performs best in fine-grained evaluation, while PN-VQA performs well against other VQA methods and Qwen2 has higher human-score correlation than other VQA models.The evaluation reports correlation of averaged element scores and element-wise accuracy.Source: §5.2 · p. 7, Table 3 · p. 7
• Across 200 representative prompts and more than 20 T2I models, proprietary models such as Midjourney and Dreamina generally perform better.The paper also evaluates MLLMs’ ability to detect structural problems; full rankings and comparisons are in the supplementary material.Source: §5.2 · pp. 7–8
• FGA-BLIP2 and PN-VQA produce fine-grained scores consistent with human annotations, while FGA-BLIP2’s overall scores align more closely than VQAScore’s.Figure 4 compares overall scores; Figure 5 shows element-match probabilities, with questions and answers based on prompt elements.Source: §5.2 · p. 8, Figure 4 · p. 8, Figure 5 · p. 8

5.3. Ablation Study

• FGA-BLIP2’s overall scores correlate better across four datasets with variance optimization than without it; combining and weighting overall and average element scores improves performance on GenAI-Bench and TIFA.The ablation compares overall scores with and without variance optimization, and tests a final score combining overall and average element scores.Source: §5.3 · p. 8
• For PN-VQA, including the image-generation prompt in questions and evaluating positive and negative questions separately improves correlation with human annotations.The ablation is reported for PN-VQA using Qwen2-VL.Source: §5.3 · p. 8, Table 4 · p. 8

• EvalMuse-40K contains manually annotated overall alignment scores and fine-grained element scores for evaluating automated metrics against human judgments.The benchmark supports evaluation across image-text alignment-related tasks.Source: §6 · p. 8
• FGA-BLIP2 fine-tunes a vision-language model to output token-level alignment scores, while PN-VQA uses positive-negative questions for fine-grained VQA.Source: §6 · p. 8
• The authors report that their methods improve correlation between image-text alignment metrics and human annotations, supporting better evaluation of alignment in T2I models.Source: §6 · pp. 8–9

• The supplement details classification and sampling of real prompts, synthetic-prompt generation, prompt-element splitting, and question generation.These topics are covered in Sections 8.1–8.5.Source: §7 · p. 11
• Additional experiments report fine-grained skill results for FGA-BLIP2 and PN-VQA, visualizations, and MLLM structural-problem detection.The overview points to Table 6 and Sections 9.1–9.3.Source: §7 · p. 11
• The supplement evaluates alignment for more than 20 widely used T2I models with FGA-BLIP2.The process and results are presented in Section 10 and Table 10.Source: §7 · p. 11

8.1. Real Prompt Categorization

• Real prompts are classified by subject, logical relationship, image style, and BERT embedding; GPT-4 labels the first three dimensions, while K-Means divides embeddings into seven types.The source prompts are sampled from 100K DiffusionDB prompts.Source: §8.1 · p. 11

8.2. Prompt Sample Strategy

• Using MILP, the method selects N=2,000 samples from K=100,000 prompts so their distribution closely aligns with target D, producing a more balanced distribution across four dimensions (Fig. 6).For each dimension m, Dm is set to Rm/Hm, where Rm is the average number of categories per prompt and Hm is the total number of categories.Source: §8.2 · pp. 11–12, Figure 6 · p. 13

8.3. Sythetic Data Generation

• Synthetic prompts target object count, color and material, setting, activity and perspective, writing symbols, and spatial composition using keyword combinations and GPT-4 templates.Source: §8.3 · pp. 12–13

8.4. Element Splitting

• GPT-4 splits prompts into elements and assigns categories including object, activity, attribute, counting, color, material, spatial, and location.The element categories follow TIFA’s categorization strategy.Source: §8.4 · p. 13

8.5. Question Generation

• GPT-4 generates simple Yes/No questions for each prompt element to check whether that element appears in the generated image.Source: §8.5 · pp. 13–14

9.1. Results of Metrics on Specific Skills

• Fine-grained metrics show relatively high accuracy across categories, and model weaknesses often correlate with evaluation weaknesses; counting is an exception, as quantity errors are harder to generate than to assess.Table 6 reports accuracy by element category.Source: §9.1 · p. 14, Table 6 · p. 15

9.2. More Qualitative Comparisons

• FGA-BLIP2 aligns more closely with human overall scores than VQAScore, while FGA-BLIP2 and PN-VQA produce fine-grained scores consistent with human annotations.Source: §9.2 · p. 14, Figure 7 · p. 15, Figure 8 · p. 15

9.3. Structural Problem Evaluation

• Structural-problem evaluation uses VQA probabilities with a separately optimized threshold per model; accuracy covers three coarse labels, while recall tests all annotated problems.Thresholds are chosen by maximizing F1-Score across positive and negative samples.Source: §9.3 · pp. 15–16
• The evaluated models achieve around 50% accuracy on object tests, and their structural-problem recall rates are relatively low.Source: Table 8 · p. 16, Table 9 · p. 16

• Proprietary models including Dreamina v2.0pro, DALLE 3, and FLUX 1.1 generally perform better in image-text alignment.Source: Table 10 · p. 17, §10 · p. 17
• The evaluation uses FGA-BLIP2 to score overall alignment and fine-grained alignment across skills for images from different models.Source: §10 · p. 17
• The test selects 200 prompts from high-variance real and synthetic prompts, with each model generating four images per prompt.The authors first choose the top 500 prompts in each group and downsample each group to 100.Source: §10 · p. 17