R2I-Bench: Benchmarking Reasoning-Driven Text-to-Image Generation

1 Introduction

• The benchmark is motivated by prompts such as a bitten apple left exposed for more than a week, which require inferring decay and composing visual context.The example illustrates why a model must reason beyond a prompt’s surface description.Source: §1 · p. 1
• Existing benchmarks often cover narrow reasoning domains, while common T2I metrics emphasize semantic alignment or do not generalize across reasoning types.The paper frames this as a gap in assessing reasoning capabilities in T2I generation.Source: §1 · p. 2
• R2I-Bench introduces 3,068 curated prompts spanning seven reasoning categories and 32 fine-grained subcategories.The benchmark is designed to evaluate reasoning capabilities in text-to-image models.Source: §1 · p. 2
• R2I-Score uses human-validated, instance-specific questions to assess text-image alignment, reasoning accuracy, and image quality.The score is reported to align strongly with human judgments.Source: §1 · p. 2
• All open-source models achieve less than 45% accuracy on R2I-Bench, and recent reasoning enhancements yield only marginal improvements.A pipeline that separates reasoning from image generation improves performance but still struggles with abstract mathematics and linguistic constructs such as quantities and quantifiers.Source: §1 · p. 2
• The paper presents R2I-Bench as a comprehensive, human-in-the-loop resource for evaluating reasoning-driven T2I generation.The comparison with existing benchmarks highlights its broader reasoning coverage and curated evaluation criteria.Source: §1 · p. 2, Table 1 · p. 2

Text-to-Image Generation Models

• Diffusion, autoregressive, and unified T2I models produce high-quality images, while newer systems incorporate chain-of-thought reasoning or reinforcement learning.The paper characterizes their reasoning capabilities as underdeveloped and insufficiently evaluated.Source: §2 · p. 3

Text-to-Image Evaluation Benchmarks and Metrics

• Existing T2I benchmarks often focus on isolated skills, including shallow or knowledge-based reasoning, numerical tasks, or compositionality.Examples include OK-VQA, WISE, Commonsense T2I, GeckoNum, Winoground, GenEval, and GenAI-Bench.Source: §2 · p. 3
• Current metrics can miss compositional, numerical, spatial, relational, or fine-grained reasoning accuracy, leaving no unified framework for evaluating the full spectrum of T2I reasoning.The passage identifies reasoning sensitivity and granularity as gaps across CLIPScore, VQAScore, LLM-based metrics, RIScore, and WIScore.Source: §2 · p. 3

Overview

• Each R2I-Bench instance includes a reasoning-based prompt, reference caption, explanation of the reasoning steps, and category metadata.The benchmark is constructed through data collection, data filtering, and evaluation-criteria generation.Source: §3 · p. 3

Data Collection

• The benchmark organizes T2I reasoning into seven core categories refined into 32 subcategories.The categories include commonsense, compositional, logical, concept mixing, numerical, mathematical, and causal reasoning.Source: §3 · p. 4
• GPT-4o generates 100–120 prompts per subcategory, guided to avoid revealing answers or directly describing visual features.Human-authored positive and negative examples guide prompt generation, and GPT-4o also produces explanation descriptions for later evaluation criteria.Source: §3 · p. 4

Data Filtering and Refinement

• Manual filtering removes prompts that cannot yield renderable images or uniquely identifiable visual elements, followed by iterative expert-guided augmentation.The process starts with about 800 instances from 3,200 and continues until each subcategory has approximately 100 validated instances.Source: §3 · p. 4

Evaluation Criteria Generation and R2I-Score

• Evaluation questions assess text-image alignment, reasoning accuracy, and image quality, with higher importance weights assigned to reasoning accuracy.Experts validate the questions, scoring criteria, and weights; the prescribed weight ranges are 7–10 for reasoning, 4–6 for alignment, and 1–3 for image quality.Source: §3 · p. 4

Dataset Statistics

• R2I-Bench contains 3,068 high-quality reasoning-driven T2I prompts, and Table 2 reports model evaluation across core reasoning categories.The table caption identifies the highest accuracies for closed-source and open-source models.Source: §3 · p. 5, Table 2 · p. 5

4.1 Experimental Setup

• The experiments evaluate 16 representative models across diffusion, autoregressive, reasoning-enhanced, and closed-source categories, using R2I-Score.The pipeline framework uses GPT-4o for prompt reasoning and rewriting, followed by SD3-medium for image generation.Source: §4.1 · p. 5

4.2 Main Results

• Most open-source models score below 45% on R2I-Score, while mathematical reasoning remains difficult across models.SD3-medium scores 0.19 on mathematical reasoning; LlamaGen, Show-o, and ELLASD1.5 each score below 0.10.Source: §4.2 · p. 6
• Reasoning-enhanced Show-o+PARM scores 0.38 overall versus 0.36 for base Show-o, with mathematical and causal scores still ≤0.13 and ≤0.32.Closed-source DALL-E-3 and gpt-image-1 score 57.8% and 71.1% higher than the best open-source model, SD3-medium; gpt-image-1 leads across categories.Source: §4.2 · p. 6
• The pipeline-based T2I framework yields average gains of 0.13 across all reasoning categories, with causal and commonsense improvements from 0.21 to 0.58, while compositional and mathematical gains remain ≤0.13.Source: §4.2 · p. 6

4.3 Evaluation of R2I-Score

• R2I-Score aligns with human judgments better than DSGscore, VIEScore, CLIPScore, and VQA score across all reported alignment criteria.Table 3 compares R2I-Score with other T2I evaluation metrics.Source: Table 3 · p. 7, §4.3 · p. 7

4.4 Error Analysis

• Reasoning-related failures account for over 80% of errors across Emu3, SD3-medium, Show-o+PARM, and gpt-image-1.The analysis groups errors into basic element errors, reasoning errors, and visual quality issues; gpt-image-1 has the lowest basic-element and image-quality error rates.Source: §4.4 · pp. 7–8, Figure 5 · p. 7

5 Conclusion

• R2I-Bench covers 7 reasoning categories and 32 subcategories, while R2I-Score is a QA-style metric with stronger human-judgment correlation than existing evaluation metrics.The evaluation finds consistently limited reasoning capabilities across existing T2I models.Source: §5 · p. 8

Limitations

• R2I-Score's evaluation questions and criteria are tied to this benchmark and cannot be directly generalized to other datasets without adaptation.The question-based evaluations are also relatively coarse compared with training-level assessments.Source: Limitations · p. 8
• Evaluation is limited to English-language image generation, excludes instances with unsupported symbolic inputs, and does not assess video, audio, or 3D generation.Non-English reasoning capabilities remain unexplored.Source: Limitations · p. 8

Ethics Statement

• Human curation and manual review mitigate risks from GPT-4o-generated instances, but residual biases may remain.The authors encourage future work to reduce bias and align generated content with ethical guidelines and societal norms.Source: Ethics Statement · p. 8

A.1 Statistics of R2I-Bench

• Table 4 presents key statistics of R2I-Bench, but the supplied passage does not state the values.Source: §A.1 · p. 11, Table 4 · p. 11

A.2 Image by Categories

• Figures 8–12 present example images from the benchmark’s reasoning categories.Source: §A.2 · p. 11

A.3 Definition of Categories in R2I-Bench

• R2I-Bench organizes instances into seven core categories and 32 more granular subcategories.Source: §A.3 · p. 11
• Commonsense uses general and contextual knowledge; compositional reasoning combines parts, while concept mixing combines semantic elements into a new concept.Source: §A.3 · p. 11
• The appendix characterizes numerical, logical, mathematical, and causal reasoning and assigns each category its own subfields.The supplied passages describe numerical reasoning as interpreting, processing, and generating images from textual numbers; logical reasoning uses premises or conditions; mathematical reasoning represents abstract concepts visually; causal reasoning concerns cause and effect.Source: §A.3 · p. 11

A.4 Definition of Subcategories in R2I-Bench

• The section presents definitions of subcategories under R2I-Bench categories, covering commonsense, numerical, causal, logical, mathematical, concept-mixing, and compositional reasoning.Source: §A.4 · p. 11

• The data-generation pipeline uses human verification at key stages to remove low-quality data, annotations, and ambiguous evaluation questions.Figure 6 depicts collection, filtering, evaluation-criteria generation, and production of R2I-Bench.Source: §A.5 · p. 11, Figure 6 · p. 12
• The appendix provides subcategory-definition and proportion tables for numerical, commonsense, mathematical, concept-mixing, causal, and compositional reasoning.The table captions identify the scope but provide no individual proportions in these passages.Source: Table 5 · p. 12, Table 6 · p. 12, Table 7 · p. 13
• Figures 10–15 illustrate examples or distributions for mathematical, concept-mixing, compositional, causal, numerical, and logical reasoning subfields.Source: Figure 10 · p. 17, Figure 11 · p. 18, Figure 12 · p. 19
• Exact Number Generation instances are restricted to values within 30 because of current visual-language-model limitations on numerical tasks.Source: Figure 16 · p. 21

B Evaluation Details

• All models are evaluated zero-shot; open-source experiments use A-40 GPUs, while closed-source experiments use provider APIs.The zero-shot setup is intended to assess generalization without few-shot prompting or additional fine-tuning.Source: B Evaluation Details · p. 22

B.1 Prompts details

• Figure 17 gives the prompt used to create data instances.Source: Figure 17 · p. 22

B.1.1 Prompt for Data Instances Creation

• GPT-4o initially generates data instances, which are then filtered for duplicates and low quality.Source: §B.1.1 · p. 22

B.1.2 Prompt for Explanation Generation

• GPT-4o generates explanations using the prompt shown in Figure 18.Source: §B.1.2 · p. 22, Figure 18 · p. 22

B.1.3 Prompt for Image Evaluation

• GPT-4o scores images against their corresponding text using the evaluation prompts after questions and criteria are refined.Figure 19 shows the image-evaluation prompt; the scoring model uses a fixed temperature of 0.1.Source: §B.1.3 · p. 22, Figure 19 · p. 22

B.1.4 Prompt for Evaluation Criteria Generation

• Evaluation questions and scoring criteria are generated with GPT-4o at temperature 0.1, separated into prompts, then reviewed and refined by human annotators.The prompts are separated because GPT-4o’s performance degrades with long contexts; Figure 20 shows the criteria-generation prompt.Source: §B.1.4 · p. 23, Figure 20 · p. 23

B.2 Human Annotators

• Senior college students compare outputs from Lumina-Image 2.0 and Sana-1.5, choosing the more prompt-aligned image or rating both equally satisfactory or unsatisfactory.Source: §B.2 · p. 23

B.3 Model Details

• The authors select the latest models and best-performing configurations for evaluation; Table 12 lists release times and MLLM sources.Source: §B.3 · p. 23

C.1 Main Results across 33 Subcategories

• Tables 13–17 report model results across subcategories in mathematical, logical, commonsense, concept-mixing, causal, numerical, and compositional reasoning.The supplied passage gives the tables’ scope but no model scores or comparisons.Source: §C.1 · p. 23

C.2 Comparison of Subcategory Performance: Standard T2I Model vs. Pipeline-based Framework

• Figure 21 presents a subcategory-performance comparison between a standard T2I model and the pipeline-based framework.The supplied passage does not state the comparison’s outcome.Source: §C.2 · p. 23

C.3 Results of Our Evaluation Methods and Additional Metrics in the Benchmark

• Table 12 lists the release time and model source for the T2I models evaluated in R2I-Bench.Source: Table 12 · p. 24
• Tables 13–17 report evaluations across mathematical, logical, commonsense, numerical, causal, concept-mixing, and compositional reasoning.Their captions identify the reasoning areas and, in several cases, the subcategories covered.Source: Table 13 · p. 25, Table 14 · p. 26, Table 15 · p. 27
• Table 18 compares the paper’s evaluation methods with other image-text alignment metrics across models and categories.The section identifies these as results of the evaluation methods and other metrics, but the supplied passages do not provide comparison outcomes.Source: §C.3 · p. 23, Table 18 · p. 30
• Figure 21 distinguishes standard T2I model results in blue from pipeline-framework changes, using green for increases and red for decreases.Source: Figure 21 · p. 31