PaperlayerView exact revision on arXiv

Source-linked AI summary

T2I-ReasonBench: Benchmarking Reasoning-Informed Text-to-Image Generation

Kaiyue Sun, Rongyao Fang, Chengqi Duan, Xian Liu, Xihui Liu

arXiv:2508.17472v1cs.CV

TL;DR

Existing text-to-image models and benchmarks leave reasoning about implicit meanings and knowledge-driven visual content insufficiently assessed. T2I-ReasonBench introduces a four-dimension benchmark and two-stage evaluation protocol, then finds that proprietary models generally reason more strongly than open-source models, although all have room to improve.

  • Problem

    Many T2I models map semantic concepts directly to visual elements, while existing benchmarks largely emphasize literal alignment rather than reasoning about implicit meanings.

    Source: §1 · p. 2, §3.1 · p. 4

  • Method

    T2I-ReasonBench evaluates four reasoning dimensions with 800 prompts and uses a two-stage protocol to assess reasoning accuracy and image quality.

    Source: §1 · p. 2

  • Results

    Open-source models have significant reasoning limitations, while proprietary models such as GPT-Image-1 and Gemini-2.0 show stronger reasoning and knowledge integration, with considerable room for improvement across models.

    Source: §1 · p. 2, §6 · p. 11

  • Takeaways & Limitations

    The benchmark and evaluation protocol provide a basis for analyzing reasoning-informed T2I generation across implicit instructions and four reasoning dimensions.

    Source: §1 · p. 2, §6 · p. 10

Abstract

from arXiv · show

We propose T2I-ReasonBench, a benchmark evaluating reasoning capabilities of text-to-image (T2I) models. It consists of four dimensions: Idiom Interpretation, Textual Image Design, Entity-Reasoning and Scientific-Reasoning. We propose a two-stage evaluation protocol to assess the reasoning accuracy and image quality. We benchmark various T2I generation models, and provide comprehensive analysis on their performances.

1 Introduction

T2I-ReasonBench targets the gap between literal prompt following and reasoning about implicit meaning. It combines a four-dimension benchmark with a two-stage evaluation framework and uses it to assess 14 T2I models.

1 Introduction

  • Existing benchmarks largely test literal alignment or composition, leaving models’ ability to reason beyond explicit instructions insufficiently evaluated.The example of a beach ball and marble in a pool requires inferring that the ball floats and the marble sinks.Source: §1 · p. 2
  • T2I-ReasonBench evaluates reasoning across Idiom Interpretation, Textual Image Design, Entity-Reasoning, and Scientific-Reasoning using 800 prompts.The dimensions address idiom meaning, design intentions, world-entity knowledge, and scientific knowledge.Source: §1 · p. 2
  • A two-stage evaluation framework uses LLM-generated question-criterion pairs and MLLM answers and scores to assess reasoning accuracy and image quality.Averaged scores produce separate Accuracy and Quality measures.Source: §1 · p. 2
  • Open-source models significantly underperform proprietary counterparts, all models have room for improvement, and unified models integrating understanding and generation show enhanced reasoning-informed generation.The evaluation covers 14 state-of-the-art models.Source: §1 · p. 2
  • The paper evaluates 14 state-of-the-art T2I models and analyzes their performance and reasoning limitations.The models include diffusion-based, auto-regressive-based, and proprietary systems.Source: §1 · p. 2

2 Related Work

The related work describes diffusion and autoregressive approaches to text-to-image generation, alongside benchmarks that increasingly test reasoning. T2I-ReasonBench adds tasks centered on interpreting idioms and designing images with incomplete specifications.

2.1 Text-to-image Generation.

  • Diffusion models progressively denoise Gaussian noise into images, while autoregressive models predict sequences of image tokens; unified architectures connect text and image modeling.Bagel can generate reasoning steps before images when its “Thinking” mode is enabled.Source: §2.1 · p. 3

2.2 Benchmarks for Text-to-image Generation.

  • Existing benchmarks span literal alignment, world knowledge, physical common sense, everyday commonsense, and reasoning across seven categories.Examples include WISE, PhyBench, Commonsense-T2I, and R2I-Bench.Source: §2.2 · p. 3
  • T2I-ReasonBench uses 800 prompts across four dimensions and adds Idiom Interpretation and Textual Image Design, where generated content may not be fully specified.These dimensions target scenario imagination and filling in missing information.Source: §2.2 · pp. 3–4
  • Figure 2 presents the prompt collection process, subcategories in the four dimensions, and prompt-suite statistics.Its panels place these materials from left to right.Source: Figure 2 · p. 4

3 Benchmark Construction

T2I-ReasonBench targets reasoning that surface-level prompt-to-image translation misses, with four dimensions spanning figurative meaning, design intent, entity knowledge, and scientific knowledge. Its 800-prompt suite operationalizes these challenges through curated and validated prompt-generation procedures.

3.1 Problem Definition

  • The benchmark defines four reasoning dimensions: Idiom Interpretation, Textual Image Design, Entity-Reasoning, and Scientific-Reasoning.They address figurative meaning, goal-oriented visual-text design, retrieval of implicit entity knowledge, and application of scientific knowledge.Source: §3.1 · p. 4

3.2 Prompt Suite of T2I-ReasonBench

  • For Idiom Interpretation, 200 everyday idioms were turned into contextual sentences that leave their figurative meanings implicit.Claude-3.5-Sonnet generated the sentences from idioms sourced from a book and the internet.Source: §3.2 · p. 5
  • Textual Image Design contains 200 prompts derived from rich-text images, each emphasizing functional design intent rather than visual details.The prompts cover infographics, posters, documents, tables, diagrams, and other real-world images.Source: §3.2 · p. 5
  • Entity-Reasoning uses 200 manually reviewed prompt-meaning pairs designed to elicit entities from contextual clues.The review checks consistency and that each entity has unique visual features.Source: §3.2 · p. 5
  • Scientific-Reasoning prompts span physics, chemistry, biology, and astronomy, with manual validation that scientific knowledge is needed and the expected image is unstated.Example: an iron ball on a trampoline implies that its weight deeply stretches the surface.Source: §3.2 · p. 5
  • Figure 3 visualizes the word distribution of each benchmark dimension, while Figure 2 presents prompt collection, subcategories, and suite statistics.The paper directs readers to the appendix for further prompt-suite information.Source: §3.2 · p. 5, Figure 3 · p. 5

4 Evaluation Metrics

The evaluation framework uses prompt-specific checks and image analysis to measure reasoning accuracy and image quality. Its reasoning-accuracy metrics correlate most strongly with human judgments across all four dimensions.

4 Evaluation Metrics

  • The framework generates prompt-specific question-criterion pairs, tailored to each prompt’s category, reasoning demands, and expected explicit and implicit content.DeepSeek-R1 [16] generates separate question sets for reasoning and image quality, plus explicit-detail questions for Entity and Scientific-Reasoning.Source: §4 · pp. 5–6
  • In the second stage, Qwen2.5-VL [2] evaluates generated images using Chain-of-Thought [43]: it describes each image, answers Stage 1 questions, analyzes each answer before scoring, and averages scores within each set into Reasoning Accuracy and Image Quality.Source: §4 · p. 6
  • Entity-Reasoning and Scientific-Reasoning final scores weight reasoning at 0.7 and other details at 0.3.The weighting prioritizes reasoning while balancing the final score.Source: §4 · p. 6

4.1 Human Evaluation Correlation Analysis

  • Reasoning Accuracy achieves the highest correlation with human judgments across all dimensions, outperforming CLIP [17] and VQA score [28].The evaluation used 400 images, with three annotators independently scoring each image; CLIP and VQA showed moderate correlations but were less effective at interpreting implicit prompt meaning.Source: §4.1 · p. 6
  • Human validation compared metric scores with averaged scores from three annotators using Kendall’s τ and Spearman’s ρ.Twenty prompts per dimension and five T2I models yielded 400 images in total.Source: §4.1 · p. 6

5 Evaluation Results

T2I-ReasonBench exposes substantial reasoning challenges for current T2I models, with open-source models generally weaker than proprietary systems. Rewriting prompts with an external LLM improves reasoning accuracy for almost all models, though gains vary by model and task.

5.1 Experimental Setup

  • The evaluation covers 14 models: seven diffusion-based, five autoregressive, and two proprietary models, using default inference settings.The framework generates prompt-specific question-criterion pairs with an LLM, then uses an MLLM to analyze and score images.Source: §5.1 · p. 6, Figure 4 · p. 7

5.2 Quantitative Evaluation

  • HiDream’s overall accuracy exceeds 50.0, but the authors report no evidence of prompt reasoning and say it still relies on concept mapping.They attribute its progress in part to using Llama 3 as a text encoder.Source: §5.2 · p. 7
  • Current T2I models, especially open-source ones, struggle to render implicit prompt meanings; Table 2 reports benchmark accuracy and quality scores.The authors describe complex prompt reasoning as a significant challenge.Source: §5.2 · p. 7, Table 2 · p. 8
  • Most evaluated AR-based models underperform diffusion-based models, while Bagel becomes comparable to diffusion models with its “Thinking” mode enabled.The authors describe Bagel’s reasoning-augmented data and unified architecture as supporting interaction between understanding and generation.Source: §5.2 · p. 8

5.3 Evaluation on Two-Stage Pipeline Setting

  • With GPT-4o rewriting prompts into visually explicit descriptions, reasoning accuracy rises for almost all models; Table 3 records results and changes from original prompts.The comparison suggests models generate appropriate scenes more effectively when intended meanings are stated clearly.Source: Table 3 · p. 9, §5.3 · p. 9
  • Rewritten prompts yield minimal improvement in Textual Image Design, where performance depends mainly on text synthesis, while clearer prompts moderately increase image quality.GPT-Image-1 improves only slightly overall, and its Textual Image Design accuracy decreases slightly.Source: §5.3 · p. 9

6 Conclusion and Discussions

T2I-ReasonBench tests implicit-meaning interpretation across four dimensions, and the evaluation finds open-source models have significant reasoning limitations. The authors call for stronger reasoning-informed T2I systems and broader reasoning-task coverage.

6 Conclusion and Discussions

  • Across four dimensions, the benchmark finds significant reasoning limitations in open-source models, while GPT-Image-1 and Gemini-2.0 show stronger reasoning and knowledge integration.The conclusion also notes considerable room for improvement in proprietary models.Source: §6 · pp. 10–11
  • Future work should incorporate structured knowledge bases and reasoning mechanisms into T2I models and broaden benchmarks to cover more reasoning tasks.The authors present these directions as ways to advance reasoning-informed T2I generation.Source: §6 · p. 11
  • The authors flag potential negative social impacts when image-generation models are used to create fake images that mislead people.Source: §6 · p. 11

A More details on prompt collection process

The benchmark’s prompts are built from selected idioms and curated text-rich images. The collection process turns these source materials into contextualized scenes and design-intention prompts.

Idiom Interpretation

  • For Idiom Interpretation, the authors manually select 200 commonly used, challenging idioms and use an LLM to turn idioms and meanings into scene descriptions with contextual clues.The generated sentences avoid directly revealing the idiom’s meaning.Source: A More details on prompt collection… · p. 15

Textual Image Design

  • Textual Image Design draws from LLAVAR-2, InfographicVQA, POSTA, and CoSyn-400k, with selected samples spanning posters, infographics, tables, diagrams, and documents.The selections include 80 LLAVAR-2 images, 40 infographics, 40 posters, and 40 CoSyn-400k samples.Source: A More details on prompt collection… · p. 15
  • Qwen2.5-VL summarizes the collected images’ design intentions, producing 200 prompts for Textual Image Design.Source: A More details on prompt collection… · p. 15

B Evaluation Framework

The evaluation framework creates prompt-specific checks and applies them to generated images. Dimension-specific templates guide question-criterion generation, while a shared template evaluates images across the four dimensions.

Evaluation Framework

  • DeepSeek-R1 generates prompt-specific question-criterion pairs, and Qwen2.5-VL evaluates generated images using those pairs.The framework focuses on reasoning accuracy and image quality.Source: B Evaluation Framework · p. 15
  • Separate templates tailor question-criterion generation to Idiom Interpretation, Textual Image Design, Entity-Reasoning, and Scientific-Reasoning.The corresponding templates are shown in Tables 4–7.Source: B Evaluation Framework · p. 15, Table 4 · p. 16, Table 5 · p. 17
  • A shared image-evaluation template applies across all four dimensions, with only the prompt-specific question-criterion pairs replaced.Table 8 also provides an Idiom Interpretation example output.Source: B Evaluation Framework · p. 15, Table 8 · p. 19

Related papers

Loading 2508.17472v1…