PaperlayerView exact revision on arXiv

Source-linked AI summary

TIIF-Bench: How Does Your T2I Model Follow Your Instructions?

Xinyu Wei, Jinrui Zhang, Zeqing Wang, Hongyang Wei, Zhen Guo, Bairui Li, Lei Zhang

arXiv:2506.02161v3cs.CV

TL;DR

Existing T2I benchmarks provide limited prompt diversity and coarse evaluation, making fine-grained instruction-following assessment difficult. TIIF-Bench addresses this with diverse, length-varied prompts, added evaluation dimensions, GNED, and a reproducible VLM evaluator; experiments reveal robustness and architecture-related patterns in current models.

  • Problem

    Existing T2I benchmarks have limited prompt diversity and complexity, while conventional evaluation protocols remain too coarse for fine-grained instruction-following assessment.

    Source: §1 · p. 3, §3.1 · pp. 6–7

  • Method

    TIIF-Bench combines attribute-composed prompts, short and extended variants, text-rendering and style-control evaluation, GNED, and attribute-specific VLM verification.

    Source: §1 · p. 4, §3.2 · p. 8, §3.3 · p. 11

  • Results

    Higher-scoring T2I models tend to be more robust to prompt-length variations, while unified multimodal approaches often achieve instruction-following performance comparable to state-of-the-art diffusion models.

    Source: §1 · p. 5

  • Takeaways & Limitations

    TIIF-Bench provides fine-grained, reproducible evaluation across diverse instruction-following dimensions and yields practical insights into current T2I systems.

    Source: §5 · p. 18

    Existing evaluation can be biased when full captions are exposed to VLMs, encouraging textual-prior reliance and overly optimistic scores for dense prompts.

    Source: §3.1 · p. 7

Abstract

from arXiv · show

The rapid advancements of Text-to-Image (T2I) models have ushered in a new phase of AI-generated content, marked by their growing ability to interpret and follow user instructions. However, existing T2I model evaluation benchmarks fall short in limited prompt diversity and complexity, as well as coarse evaluation metrics, making it difficult to evaluate the fine-grained alignment performance between textual instructions and generated images. In this paper, we present TIIF-Bench Text-to-Image Instruction Following Benchmark), aiming to systematically assess T2I models' ability in interpreting and following intricate textual instructions. TIIF-Bench comprises 5,000 prompts organized along multiple dimensions and categorized into three levels of difficulty and complexity. To rigorously evaluate robustness to prompt length, each prompt is provided in both short and long versions with identical core semantics. We further propose a novel Global Normalized Edit Distance (GNED) metric for text rendering and provide aspect-ratio-diverse reference images for each prompt to assess style control. In addition, we collect 100 high-quality designer-level prompts covering diverse scenarios for comprehensive evaluation. To enable scalable and fine-grained evaluation, we explore the best paradigm for leveraging the world knowledge encoded in large Vision-Language Models (VLMs) as automated binary evaluators. Through extensive ablations, we develop a fully reproducible evaluator that provides interpretable reasoning and reliable verification, enabling our benchmark to discern subtle variations in T2I model outputs. Through comprehensive benchmarking of mainstream T2I models on TIIF-Bench, we analyze the strengths and weaknesses of current T2I systems and reveal the limitations of existing evaluation benchmarks. Project Page: https://a113n-w3i.github.io/TIIF_Bench/.

1 Introduction

TIIF-Bench addresses gaps in T2I evaluation by diversifying prompts across instruction-following dimensions, varying prompt length, and enabling fine-grained VLM-based verification. Benchmarking reveals prompt-length robustness varies across models and that unified multimodal architectures can follow instructions comparably despite lower visual fidelity.

Motivation

  • Existing benchmarks often use fixed-length, redundant prompts and coarse scorers, limiting fine-grained evaluation of instruction following.These limitations include inadequate prompt-length robustness testing, semantic redundancy, and disagreement between expert scores and human judgments.Source: §1 · p. 3, Figure 3 · p. 4

Benchmark Design

  • TIIF-Bench constructs semantically diverse prompts across composition, reasoning, text rendering, and style control, with short and extended versions for each prompt.The benchmark combines ten concept pools with six compositional dimensions, adds two evaluation dimensions, and includes 100 designer-level prompts within 5,000 total prompts.Source: §1 · pp. 4–5

Findings

  • Higher-scoring models tend to be more robust to prompt-length changes, while unified multimodal approaches often match instruction-following performance of stronger-fidelity diffusion models.The paper links this pattern to multimodal reasoning within a unified transformer backbone, while presenting it as an observed correlation and architectural advantage.Source: §1 · p. 5

Benchmark Design

  • Attribute-composition sampling improves semantic coverage and syntactic diversity while systematically testing robustness to prompt length.The methodology targets fixed-length prompts, semantic redundancy, and limited syntactic diversity in existing benchmarks.Source: §1 · p. 5

Evaluation

  • TIIF-Evaluator uses attribute-specific yes/no verification and interpretable reasoning to assess alignment more finely and reliably than coarse metrics.Each prompt is paired with a checklist of binary questions for VLM-based verification.Source: §1 · pp. 4–5

2 Related Work

Prior T2I evaluation uses preference-aligned scoring models and structured benchmarks, but both face limitations in capturing fine-grained semantic correspondence. VLM-based approaches broaden evaluation but commonly remain holistic rather than attribute-specific.

Evaluation Streams

  • T2I evaluation research comprises preference-aligned scoring methods and benchmark-driven structured evaluation frameworks.Benchmark-driven methods organize prompts around compositional dimensions such as attributes, relations, and numeracy.Source: §2 · pp. 5–6

Preference-Aligned Scoring

  • CLIP-based scoring methods improve alignment with human judgments in some settings but remain constrained in fine-grained semantic correspondence.CLIPScore can fail on complex prompts, while HPSv2 improves alignment with human judgments through preference-data mitigation.Source: §2 · p. 5
  • VisionReward uses hierarchical visual assessment and interpretable weighting, but its primary design for video quality limits applicability to static T2I tasks.Source: §2 · pp. 5–6

VLM-Based Evaluation

  • UnifiedReward methods use VLMs for overall image-text alignment but typically cannot verify specific semantic attributes or binary questions.Their holistic scores do not provide the fine-grained verification required for detailed instruction-following assessment.Source: §2 · p. 6

Benchmark-Driven Evaluation

  • Benchmark-driven frameworks commonly rely on CLIP-based metrics or relatively weak VQA models for automatic evaluation.Examples include TIFA, DSG-Bench, DPG-Bench, Gecko, CompBench++, GenEval, and GenAI Bench.Source: §2 · p. 6

3 Construction of TIIF-Bench

TIIF-Bench addresses weaknesses in existing T2I benchmarks by diversifying prompt construction, adding advanced instruction-following dimensions, and introducing fine-grained VLM-based evaluation. Its pipeline combines attribute composition, length-preserving expansion, specialized prompts, and attribute-level verification.

Limitations of Current T2I Evaluation Benchmarks

  • Existing benchmarks limit evaluation through fixed-length, semantically redundant prompts and coarse protocols that rely on weak scorers or generic VLM queries.Exposing the full prompt can encourage VLMs to rely on textual priors, hallucinate missing attributes or relations, and assign overly optimistic scores.Source: §3.1 · pp. 6–7

Prompt Construction

  • TIIF-Bench constructs prompts through concept pools and attribute composition to improve semantic uniqueness, compositional diversity, and sentence-structure variation.The process extracts object–attribute/relation pairs, forms ten concept pools, and combines attributes using 36 composition patterns.Source: §3.2 · pp. 7–8
  • The benchmark adds text-rendering, style-control, and designer-level dimensions to assess capabilities beyond conventional compositional reasoning.Style-control prompts use reference images with multiple aspect ratios, while designer-level prompts encode practical constraints and domain-specific knowledge.Source: §3.2 · pp. 8–9, Figure 5 · p. 9, §3.3 · p. 10
  • Each generated prompt receives a long-form counterpart through paraphrasing and elaboration that preserves the original semantics, enabling prompt-length robustness evaluation.The pipeline also reuses existing benchmark prompts and incorporates Lex-Art text-rendering prompts, curated style-control prompts, and real-world designer-level prompts.Source: §3.2 · pp. 7, 9, Figure 5 · p. 9

Evaluation Method

  • The TIIF Evaluator decomposes prompts into attribute-specific yes/no questions and uses VLM world knowledge with interpretable reasoning for fine-grained alignment verification.For designer-level prompts, manually constructed tailored question lists address complex real-world constraints and human priors.Source: §3.3 · p. 10

Ablation and Training of TIIF Evaluator

  • 91.43% accuracy was achieved by representation-based classification, while reasoning-based prediction reached 91.07% and was selected for interpretable evaluator construction.Direct prediction achieved 89.46% accuracy; both latter approaches met the reliability requirement, but reasoning-based prediction exposed instruction-following failures through chain-of-thought explanations.Source: §3.4 · p. 13

4 Experiments

TIIF-Bench evaluates diverse T2I systems across difficulty levels, prompt lengths, rendering, style, and designer-level instructions. Results show that stronger instruction following generally coincides with visual quality and prompt-length robustness, while existing benchmarks can poorly discriminate models or align with human judgment.

Evaluation Setup

  • TIIF-Bench reports average scores across basic, advanced, and designer-level prompts under short and long settings, with overall averages across dimensions.GNED values are transformed as 1 − GNED so higher Text scores consistently indicate better performance.Source: §4.1 · p. 13

Diffusion-Based Open-Source Models

  • Qwen-Image and FLUX.2 Dev show the strongest instruction-following performance among diffusion-based open-source models.LuminaImage2 and FLUX.1 Dev closely follow, while MMDiT-based models dominate high-quality image generation in this group.Source: §4.1 · p. 13

Prompt-Length Robustness

  • Top-performing models remain robust across semantically equivalent short and long prompts, whereas weaker models show substantial performance discrepancies.The results associate accurate interpretation of complex instructions with overall image-generation capability.Source: §4.1 · p. 13, Closed-Source Models · pp. 14–15

Unified Open-Source Models

  • Emu3.5 achieves the strongest overall performance among unified open-source models, but most unified models are weaker than similarly scaled diffusion models at rendering text.Emu3.5 is an exception and also demonstrates strong style control.Source: Closed-Source Models · p. 15

Closed-Source Models

  • Nano-Banana and GPT-Image-1 outperform other closed-source models across instruction-following dimensions, style control, designer-level prompts, and prompt-length robustness.Their advantage is reported as likely related to unified architectures, stronger language understanding, and large-scale high-quality training data.Source: Diffusion-based vs. AR-based… · pp. 15–16

Comparison with Existing Benchmarks

  • Existing benchmarks provide limited fine-grained discrimination and unstable human alignment, whereas TIIF-Bench achieves consistently high agreement with blinded human preferences.GenAI Bench produces highly similar VQAScore values across models; CompBench++ can assign identical scores, disagree across evaluator types, and show weak or unstable Spearman alignment.Source: §4.2 · pp. 16–18

5 Conclusion

TIIF-Bench is presented as a hierarchical benchmark for fine-grained T2I instruction following, combining diverse prompt construction, expanded evaluation dimensions, and reproducible VLM-based assessment. The authors report insights into complex instruction following while identifying remaining scope limitations.

Conclusion

  • TIIF-Bench evaluates T2I instruction following across diverse concept combinations and dimensions including text rendering, style control, and designer-level prompts.The benchmark uses dimensions grouped into Attributes, Relations, and Reasoning, with concepts mined and reorganized from prior benchmarks.Source: §5 · p. 18, Table A1 · p. 19
  • The benchmark uses a reproducible reasoning-based TIIF Evaluator for fine-grained assessment and proposes GNED for text rendering quality.It also supplies aspect-ratio-diverse reference images for style-control prompts.Source: §5 · p. 18
  • The authors report that extensive experiments reveal patterns in how current T2I models follow complex instructions and provide insights for future research.Source: §5 · p. 18

Limitations

  • The benchmark mainly uses common objects, currently supports only English prompts, and does not examine stylistic variations such as formal versus conversational wording.Future work proposes obscure vocabulary, rare objects, linguistic diversity, and stylistic-variation studies.Source: §5 · pp. 18–19

Conclusion

  • TIIF-Bench’s prompt construction balances controllable composition with semantic fidelity to source benchmarks.CompBench++ concepts are decomposed into components, whereas tightly coupled GenAI Bench structures are retained as whole phrases.Source: Table A1 · p. 19

A1 Benchmark Statistics

TIIF-Bench organizes prompts through a ten-dimension taxonomy and reusable concept composition, while reporting prompt and checklist statistics to characterize linguistic diversity and evaluation granularity. Short and long prompts preserve the same semantic target and share checklists.

Benchmark Taxonomy

  • The ten benchmark dimensions are grouped into Attributes, Relations, and Reasoning, covering progressively harder instruction-following abilities.The taxonomy ranges from local attribute binding to relational understanding and higher-level reasoning.Source: A1.1 Details of the Concept Pools · p. 19

Concept Pools

Prompt Composition

Statistics

A2 Meta Prompts

The appendix specifies meta prompts for length augmentation and image evaluation. These prompts preserve visual semantics during expansion and require grounded binary judgments with explicit reasoning and output formatting.

A2.1 Meta Prompt for Length Augmentation

A2.2 Meta Prompt for Evaluation

A3 Additional Experimental Results

Additional experiments develop and validate TIIF-Evaluator as a scalable, fine-grained VLM evaluator. The evidence indicates stronger alignment with human judgments across heterogeneous T2I systems than direct use of a general-purpose VLM.

Evaluator Design

Training Corpus

Validation

Training Paradigms

Additional Validation

A4 Additional Visualizations

Additional visualizations examine model behavior across spatial relations, style control, text rendering, designer-level prompts, evaluator agreement, and human preference. Together, they show failures in spatial reasoning, prompt-sensitive style adherence, and the broader scope of instruction-following evaluation.

Spatial Relations

  • Strong models show strong instruction following for object attributes such as color and material, but performance degrades on spatial relations.Figure A1 illustrates failure cases from strong closed-source models on spatial-relation prompts.Source: A4.1 Failure Cases of Strong Models in… · p. 24

Style Control

  • Style control evaluates global artistic directives while preserving semantic consistency across medium, palette, texture, composition, and aesthetic coherence.Reference images spanning 16:9, 5:4, 1:1, 4:5, and 9:16 reduce layout-related mismatches during comparison.Source: A4.2 Additional Examples of Style… · p. 24
  • Style adherence varies by model and prompt formulation: some models benefit from richer long descriptions, while others prefer short, label-like style cues.Strong models can reproduce global visual identity, whereas weaker models often preserve semantics while only partially matching the intended style.Source: A4.2 Additional Examples of Style… · pp. 24–25

Text Rendering

  • Text rendering assesses embedded language text using OCR Recall and the proposed GNED metric across qualitative model examples.The dimension targets complex, non-natural textures represented by human language embedded in images.Source: A4.3 Additional Examples of Text… · p. 25

Designer-level Prompts

  • Designer-level prompts provide the most comprehensive test by combining dense and diverse requirements across scenarios.Figure A5 presents qualitative examples of current T2I models on this evaluation dimension.Source: A4.4 Additional Examples of Real-world… · p. 25

AR-based versus Diffusion-based Models

  • AR-based models may have lower visual fidelity yet exhibit strong instruction following, with Janus-Pro outperforming PixArt-Sigma especially on differentiation, comparison, and negation.The comparison links autoregressive training on generation and understanding tasks with strong performance on reasoning-logic prompts.Source: A4.5 AR-based versus Diffusion-based… · p. 25

Related papers

Loading 2506.02161v3…