PaperlayerView exact revision on arXiv

Source-linked AI summary

TIIF-Bench: How Does Your T2I Model Follow Your Instructions?

Xinyu Wei, Jinrui Zhang, Zeqing Wang, Hongyang Wei, Zhen Guo, Bairui Li, Lei Zhang

arXiv:2506.02161v3cs.CV

TL;DR

Existing T2I benchmarks provide limited prompt diversity and coarse evaluation, making fine-grained instruction-following assessment difficult. TIIF-Bench addresses this with diverse, length-varied prompts, added evaluation dimensions, GNED, and a reproducible VLM evaluator; experiments reveal robustness and architecture-related patterns in current models.

  • Problem

    Existing T2I benchmarks have limited prompt diversity and complexity, while conventional evaluation protocols remain too coarse for fine-grained instruction-following assessment.

  • Method

    TIIF-Bench combines attribute-composed prompts, short and extended variants, text-rendering and style-control evaluation, GNED, and attribute-specific VLM verification.

  • Results

    Higher-scoring T2I models tend to be more robust to prompt-length variations, while unified multimodal approaches often achieve instruction-following performance comparable to state-of-the-art diffusion models.

  • Takeaways & Limitations

    TIIF-Bench provides fine-grained, reproducible evaluation across diverse instruction-following dimensions and yields practical insights into current T2I systems.

  • Takeaways & Limitations

    Existing evaluation can be biased when full captions are exposed to VLMs, encouraging textual-prior reliance and overly optimistic scores for dense prompts.

Abstract

from arXiv · show

The rapid advancements of Text-to-Image (T2I) models have ushered in a new phase of AI-generated content, marked by their growing ability to interpret and follow user instructions. However, existing T2I model evaluation benchmarks fall short in limited prompt diversity and complexity, as well as coarse evaluation metrics, making it difficult to evaluate the fine-grained alignment performance between textual instructions and generated images. In this paper, we present TIIF-Bench Text-to-Image Instruction Following Benchmark), aiming to systematically assess T2I models' ability in interpreting and following intricate textual instructions. TIIF-Bench comprises 5,000 prompts organized along multiple dimensions and categorized into three levels of difficulty and complexity. To rigorously evaluate robustness to prompt length, each prompt is provided in both short and long versions with identical core semantics. We further propose a novel Global Normalized Edit Distance (GNED) metric for text rendering and provide aspect-ratio-diverse reference images for each prompt to assess style control. In addition, we collect 100 high-quality designer-level prompts covering diverse scenarios for comprehensive evaluation. To enable scalable and fine-grained evaluation, we explore the best paradigm for leveraging the world knowledge encoded in large Vision-Language Models (VLMs) as automated binary evaluators. Through extensive ablations, we develop a fully reproducible evaluator that provides interpretable reasoning and reliable verification, enabling our benchmark to discern subtle variations in T2I model outputs. Through comprehensive benchmarking of mainstream T2I models on TIIF-Bench, we analyze the strengths and weaknesses of current T2I systems and reveal the limitations of existing evaluation benchmarks. Project Page: https://a113n-w3i.github.io/TIIF_Bench/.

1 Introduction

TIIF-Bench addresses gaps in T2I evaluation by diversifying prompts across instruction-following dimensions, varying prompt length, and enabling fine-grained VLM-based verification. Benchmarking reveals prompt-length robustness varies across models and that unified multimodal architectures can follow instructions comparably despite lower visual fidelity.

  • Motivation: Existing benchmarks often use fixed-length, redundant prompts and coarse scorers, limiting fine-grained evaluation of instruction following.These limitations include inadequate prompt-length robustness testing, semantic redundancy, and disagreement between expert scores and human judgments.
  • Benchmark Design: TIIF-Bench constructs semantically diverse prompts across composition, reasoning, text rendering, and style control, with short and extended versions for each prompt.The benchmark combines ten concept pools with six compositional dimensions, adds two evaluation dimensions, and includes 100 designer-level prompts within 5,000 total prompts.
  • Findings: Higher-scoring models tend to be more robust to prompt-length changes, while unified multimodal approaches often match instruction-following performance of stronger-fidelity diffusion models.The paper links this pattern to multimodal reasoning within a unified transformer backbone, while presenting it as an observed correlation and architectural advantage.
  • Benchmark Design: Attribute-composition sampling improves semantic coverage and syntactic diversity while systematically testing robustness to prompt length.The methodology targets fixed-length prompts, semantic redundancy, and limited syntactic diversity in existing benchmarks.
  • Evaluation: TIIF-Evaluator uses attribute-specific yes/no verification and interpretable reasoning to assess alignment more finely and reliably than coarse metrics.Each prompt is paired with a checklist of binary questions for VLM-based verification.

2 Related Work

Prior T2I evaluation uses preference-aligned scoring models and structured benchmarks, but both face limitations in capturing fine-grained semantic correspondence. VLM-based approaches broaden evaluation but commonly remain holistic rather than attribute-specific.

  • Evaluation Streams: T2I evaluation research comprises preference-aligned scoring methods and benchmark-driven structured evaluation frameworks.Benchmark-driven methods organize prompts around compositional dimensions such as attributes, relations, and numeracy.
  • Preference-Aligned Scoring: CLIP-based scoring methods improve alignment with human judgments in some settings but remain constrained in fine-grained semantic correspondence.CLIPScore can fail on complex prompts, while HPSv2 improves alignment with human judgments through preference-data mitigation.
  • Preference-Aligned Scoring: VisionReward uses hierarchical visual assessment and interpretable weighting, but its primary design for video quality limits applicability to static T2I tasks.
  • VLM-Based Evaluation: UnifiedReward methods use VLMs for overall image-text alignment but typically cannot verify specific semantic attributes or binary questions.Their holistic scores do not provide the fine-grained verification required for detailed instruction-following assessment.
  • Benchmark-Driven Evaluation: Benchmark-driven frameworks commonly rely on CLIP-based metrics or relatively weak VQA models for automatic evaluation.Examples include TIFA, DSG-Bench, DPG-Bench, Gecko, CompBench++, GenEval, and GenAI Bench.

3 Construction of TIIF-Bench

TIIF-Bench addresses weaknesses in existing T2I benchmarks by diversifying prompt construction, adding advanced instruction-following dimensions, and introducing fine-grained VLM-based evaluation. Its pipeline combines attribute composition, length-preserving expansion, specialized prompts, and attribute-level verification.

  • Limitations of Current T2I Evaluation Benchmarks: Existing benchmarks limit evaluation through fixed-length, semantically redundant prompts and coarse protocols that rely on weak scorers or generic VLM queries.Exposing the full prompt can encourage VLMs to rely on textual priors, hallucinate missing attributes or relations, and assign overly optimistic scores.
  • Prompt Construction: TIIF-Bench constructs prompts through concept pools and attribute composition to improve semantic uniqueness, compositional diversity, and sentence-structure variation.The process extracts object–attribute/relation pairs, forms ten concept pools, and combines attributes using 36 composition patterns.
  • Prompt Construction: The benchmark adds text-rendering, style-control, and designer-level dimensions to assess capabilities beyond conventional compositional reasoning.Style-control prompts use reference images with multiple aspect ratios, while designer-level prompts encode practical constraints and domain-specific knowledge.
  • Prompt Construction: Each generated prompt receives a long-form counterpart through paraphrasing and elaboration that preserves the original semantics, enabling prompt-length robustness evaluation.The pipeline also reuses existing benchmark prompts and incorporates Lex-Art text-rendering prompts, curated style-control prompts, and real-world designer-level prompts.
  • Evaluation Method: The TIIF Evaluator decomposes prompts into attribute-specific yes/no questions and uses VLM world knowledge with interpretable reasoning for fine-grained alignment verification.For designer-level prompts, manually constructed tailored question lists address complex real-world constraints and human priors.
  • Ablation and Training of TIIF Evaluator: 91.43% accuracy was achieved by representation-based classification, while reasoning-based prediction reached 91.07% and was selected for interpretable evaluator construction.Direct prediction achieved 89.46% accuracy; both latter approaches met the reliability requirement, but reasoning-based prediction exposed instruction-following failures through chain-of-thought explanations.

4 Experiments

TIIF-Bench evaluates diverse T2I systems across difficulty levels, prompt lengths, rendering, style, and designer-level instructions. Results show that stronger instruction following generally coincides with visual quality and prompt-length robustness, while existing benchmarks can poorly discriminate models or align with human judgment.

  • Evaluation Setup: TIIF-Bench reports average scores across basic, advanced, and designer-level prompts under short and long settings, with overall averages across dimensions.GNED values are transformed as 1 − GNED so higher Text scores consistently indicate better performance.
  • Diffusion-Based Open-Source Models: Qwen-Image and FLUX.2 Dev show the strongest instruction-following performance among diffusion-based open-source models.LuminaImage2 and FLUX.1 Dev closely follow, while MMDiT-based models dominate high-quality image generation in this group.
  • Prompt-Length Robustness: Top-performing models remain robust across semantically equivalent short and long prompts, whereas weaker models show substantial performance discrepancies.The results associate accurate interpretation of complex instructions with overall image-generation capability.
  • Unified Open-Source Models: Emu3.5 achieves the strongest overall performance among unified open-source models, but most unified models are weaker than similarly scaled diffusion models at rendering text.Emu3.5 is an exception and also demonstrates strong style control.
  • Closed-Source Models: Nano-Banana and GPT-Image-1 outperform other closed-source models across instruction-following dimensions, style control, designer-level prompts, and prompt-length robustness.Their advantage is reported as likely related to unified architectures, stronger language understanding, and large-scale high-quality training data.
  • Comparison with Existing Benchmarks: Existing benchmarks provide limited fine-grained discrimination and unstable human alignment, whereas TIIF-Bench achieves consistently high agreement with blinded human preferences.GenAI Bench produces highly similar VQAScore values across models; CompBench++ can assign identical scores, disagree across evaluator types, and show weak or unstable Spearman alignment.

5 Conclusion

TIIF-Bench is presented as a hierarchical benchmark for fine-grained T2I instruction following, combining diverse prompt construction, expanded evaluation dimensions, and reproducible VLM-based assessment. The authors report insights into complex instruction following while identifying remaining scope limitations.

  • Conclusion: TIIF-Bench evaluates T2I instruction following across diverse concept combinations and dimensions including text rendering, style control, and designer-level prompts.The benchmark uses dimensions grouped into Attributes, Relations, and Reasoning, with concepts mined and reorganized from prior benchmarks.
  • Conclusion: The benchmark uses a reproducible reasoning-based TIIF Evaluator for fine-grained assessment and proposes GNED for text rendering quality.It also supplies aspect-ratio-diverse reference images for style-control prompts.
  • Conclusion: The authors report that extensive experiments reveal patterns in how current T2I models follow complex instructions and provide insights for future research.
  • Limitations: The benchmark mainly uses common objects, currently supports only English prompts, and does not examine stylistic variations such as formal versus conversational wording.Future work proposes obscure vocabulary, rare objects, linguistic diversity, and stylistic-variation studies.
  • Conclusion: TIIF-Bench’s prompt construction balances controllable composition with semantic fidelity to source benchmarks.CompBench++ concepts are decomposed into components, whereas tightly coupled GenAI Bench structures are retained as whole phrases.

A1 Benchmark Statistics

TIIF-Bench organizes prompts through a ten-dimension taxonomy and reusable concept composition, while reporting prompt and checklist statistics to characterize linguistic diversity and evaluation granularity. Short and long prompts preserve the same semantic target and share checklists.

  • Benchmark Taxonomy: The ten benchmark dimensions are grouped into Attributes, Relations, and Reasoning, covering progressively harder instruction-following abilities.The taxonomy ranges from local attribute binding to relational understanding and higher-level reasoning.
  • Concept Pools: Concept pools are mined from existing T2I benchmarks and reorganized into a reusable composition space.This provides source material for constructing new benchmark prompts.
  • Concept Pools: CompBench++ concepts are decomposed into object-centered units with associated attributes or relations for flexible recombination.
  • Concept Pools: GenAI Bench prompts containing comparison, negation, or differentiation are preserved as whole phrases because their semantics are tightly coupled.
  • Prompt Composition: The composition design balances controllability with semantic fidelity, supporting diversity and interpretability in the resulting prompt set.
  • Statistics: Prompt length is measured in words, while checklist length is the average number of binary verification questions per prompt dimension.
  • Statistics: Short and long versions share the same semantic target and therefore use the same checklist, with one checklist statistic reported per dimension.
  • Statistics: The statistics characterize diversity in prompt formulation and variability in evaluation granularity for diagnosing behavior under linguistic and task complexity.

A2 Meta Prompts

The appendix specifies meta prompts for length augmentation and image evaluation. These prompts preserve visual semantics during expansion and require grounded binary judgments with explicit reasoning and output formatting.

  • A2.1 Meta Prompt for Length Augmentation: GPT-4o generates a longer version of each prompt by paraphrasing and elaborating while preserving its original meaning.
  • A2.1 Meta Prompt for Length Augmentation: The length-augmentation instruction permits richer syntax and stylistic detail but forbids introducing new objects, entities, or events.
  • A2.2 Meta Prompt for Evaluation: Each generated prompt-image pair is evaluated by inserting its yes/no questions into a predefined meta prompt sent to TIIF-Evaluator.
  • A2.2 Meta Prompt for Evaluation: The evaluator receives an image and a binary question and answers using only the image’s visual content.
  • A2.2 Meta Prompt for Evaluation: Evaluation rules require judgments grounded in visible evidence and attention to objects, attributes, counts, relations, actions, and negation.
  • A2.2 Meta Prompt for Evaluation: When evidence is ambiguous or weak, the evaluator must explain that uncertainty while still providing the most likely binary answer.
  • A2.2 Meta Prompt for Evaluation: The required response format places step-by-step visual reasoning before a final answer of exactly Yes or No.

A3 Additional Experimental Results

Additional experiments develop and validate TIIF-Evaluator as a scalable, fine-grained VLM evaluator. The evidence indicates stronger alignment with human judgments across heterogeneous T2I systems than direct use of a general-purpose VLM.

  • Evaluator Design: TIIF-Evaluator is explicitly trained for attribute-level binary verification under the TIIF-Bench protocol, unlike generic or weak-scoring evaluation pipelines.
  • Training Corpus: 30 T2I models were rolled out twice on 5,000 prompts, producing approximately 300K images and 724K binary questions for evaluator training.
  • Validation: A held-out validation set contains 300 manually annotated images and 560 balanced binary questions, split evenly between positive and negative labels.
  • Validation: GPT-4o achieved 90.18% accuracy on the held-out set, compared with 84.11% for off-the-shelf Qwen3VL-8B.
  • Training Paradigms: The evaluator study compares direct prediction, representation-based classification, and reasoning-based prediction for Qwen3VL-8B.Each paradigm was trained for one epoch.
  • Additional Validation: Additional validation covers eight T2I models spanning diverse generation quality and instruction-following capability.
  • Additional Validation: Professional annotators manually answered checklist questions for one prompt from each of nine dimensions per model to obtain human labels.
  • Additional Validation: TIIF-Evaluator remains better aligned with human judgments across diverse T2I systems, rather than only improving average accuracy on one held-out set.

A4 Additional Visualizations

Additional visualizations examine model behavior across spatial relations, style control, text rendering, designer-level prompts, evaluator agreement, and human preference. Together, they show failures in spatial reasoning, prompt-sensitive style adherence, and the broader scope of instruction-following evaluation.

  • Spatial Relations: Strong models show strong instruction following for object attributes such as color and material, but performance degrades on spatial relations.Figure A1 illustrates failure cases from strong closed-source models on spatial-relation prompts.
  • Style Control: Style control evaluates global artistic directives while preserving semantic consistency across medium, palette, texture, composition, and aesthetic coherence.Reference images spanning 16:9, 5:4, 1:1, 4:5, and 9:16 reduce layout-related mismatches during comparison.
  • Style Control: Style adherence varies by model and prompt formulation: some models benefit from richer long descriptions, while others prefer short, label-like style cues.Strong models can reproduce global visual identity, whereas weaker models often preserve semantics while only partially matching the intended style.
  • Text Rendering: Text rendering assesses embedded language text using OCR Recall and the proposed GNED metric across qualitative model examples.The dimension targets complex, non-natural textures represented by human language embedded in images.
  • Designer-level Prompts: Designer-level prompts provide the most comprehensive test by combining dense and diverse requirements across scenarios.Figure A5 presents qualitative examples of current T2I models on this evaluation dimension.
  • AR-based versus Diffusion-based Models: AR-based models may have lower visual fidelity yet exhibit strong instruction following, with Janus-Pro outperforming PixArt-Sigma especially on differentiation, comparison, and negation.The comparison links autoregressive training on generation and understanding tasks with strong performance on reasoning-logic prompts.
Loading 2506.02161v3…