Source-linked AI summary
Imag-Eval: a language-grounded framework for interpretable Text-to-Image instruction following evaluation
Ibrahim Mohamed Serouis, David Jaramillo Duque
TL;DR
Existing T2I evaluations often miss global incoherence and confound compositional difficulty with surface prompt properties. Imag-Eval independently varies instances and constraints in a controlled benchmark, finding that structured-skill performance is primarily driven by grounded rules and their binding to instances rather than prompt length.
Problem
Existing T2I evaluations are difficult to interpret, overlook global incoherence and incomplete objects, and often control difficulty through prompt length or other single-axis proxies.
Method
Imag-Eval independently varies the number of instances and combinations of constraints while evaluating compositional instruction following with a cohesiveness dimension and controlled compositional load.
Results
For structured skills, model performance is primarily driven by compositional load—the number of grounded rules and their binding to instances—rather than surface linguistic properties such as prompt length.
Takeaways & Limitations
Controlled benchmarks that disentangle linguistic complexity from compositional structure provide more interpretable and diagnostic assessment of T2I instruction following.
Takeaways & Limitations
All Imag-Eval prompts are in English, so findings may not directly generalize to languages with substantially different typological, morphological, or syntactic properties.
Abstract
from arXiv · showhide
Text-to-Image (T2I) models have recently achieved impressive visual fidelity, yet their evaluation remains constrained by benchmarks that are often difficult to interpret and insufficiently diagnostic. Existing skill-based evaluations tend to overlook critical failure modes that strongly impact usability but fall outside standard taxonomies, such as global incoherence arising from missing parts or physically implausible configurations (e.g., floating objects). In addition, prompt difficulty is typically controlled along a single dimension; either prompt length or the number of elements to generate. To address these limitations, we introduce Imag-Eval, a controlled benchmark designed to assess how T2I models ground compositional natural-language instructions into visual outputs. Unlike prior work that conflates surface linguistic complexity with compositional difficulty, Imag-Eval explicitly seeks to disentangles these factors by independently varying both the number of instances and the combination of constraints (rules), while avoiding error propagation. This design enables fine-grained and interpretable analysis of where cross-modal instruction following fails. Our benchmark comprises 1,140 prompts and 8,842 combined rules, and we evaluate it on several state-of-the-art models. Complementing this analysis with an additional study of over 2,000 prompts from a concurrent benchmark, our results suggest that, for structured skills, compositional difficulty is primarily governed by the number of grounded rules and their binding to instances,, rather than by prompt length alone.
1 Introduction
Imag-Eval addresses limitations in T2I evaluation by separating surface linguistic complexity from compositional difficulty and by exposing coherence failures. Its controlled benchmark varies instances and constraints independently, supporting interpretable analysis of instruction-following failures.
- Existing evaluations provide limited diagnostic insight and often miss globally incoherent scenes, such as missing parts or inconsistent configurations.They also commonly control difficulty through prompt length or isolated skill hardness rather than compositional structure.
- Imag-Eval varies the number of instances and combinations of constraints as orthogonal factors of compositional load.This design distinguishes surface linguistic complexity from the underlying difficulty of grounding multiple requirements.
- The framework adds coherent and complete object generation as an evaluation dimension for failures overlooked by standard skill taxonomies.The dimension targets global incoherence and incomplete objects.
- 1,140 prompts and 8,842 evaluation rules form the benchmark, validated with 14 annotators, over 8,000 annotations, and more than 6,000 generated images.
- For structured skills, performance is primarily driven by compositional load and rule-to-instance binding rather than prompt length.An additional analysis covered more than 2,000 prompts from a concurrent benchmark.
2 Related Work
Prior T2I evaluation often sacrifices diagnostic clarity by using aggregate or single-axis measures and by overlooking coherence across interacting constraints. Imag-Eval positions controlled factorization and compositional structure as complementary tools for more interpretable assessment.
- XAI-oriented evaluation decomposes performance into meaningful dimensions to localize errors beyond aggregate similarity scores.This is especially important for generative multimodal systems, where high-level metrics can conflate distinct behaviors.
- Diffusion-based T2I systems have improved fidelity and controllability, but evaluation has not kept pace with compositional instruction following.
- Earlier benchmarks often assessed skills in isolation, while later benchmarks added multiple constraints but still provided limited coverage of global coherence.
- Prompt length can confound evaluation, making it an incomplete proxy for difficulty when it varies independently of compositional structure.
- Evaluation pipelines may propagate upstream errors into downstream failures and underrepresent missing parts or physically implausible configurations.Specialized artifact evaluations do not provide a general, model-agnostic coherence framework across entities.
- Imag-Eval independently varies instances and constraint combinations to isolate failure modes without confounding error propagation.It complements existing benchmarks with a diagnostic perspective centered on compositional structure.
3 Proposed method: Imag-Eval
Imag-Eval operationalizes T2I instruction following through skill-specific rules, a cohesiveness measure, controlled compositional load, and semi-modular prompt generation. Its benchmark spans varied skill combinations, difficulty levels, perturbations, and model families.
- Common evaluation skills: Counting checks whether the generated number of object instances matches the prompt, scoring successful object cases against the total.
- Common evaluation skills: Spatial Relationships evaluates relative object positioning, while Color and Size assess instance colors and relative size constraints.
- Common evaluation skills: Emotion scores whether each character expresses the specified emotion, and Text measures how closely generated text matches the prompt.
- Under-explored evaluation skill: Cohesiveness is false when any generated object lacks essential parts or coherence, including unprompted anatomical, structural, or physical inconsistencies.The measure captures cases where other criteria are satisfied but the image remains unusable because of global incoherence.
- Compositional load: Compositional load sums instantiated objects and simultaneously satisfied constraints, supporting comparisons from two-skill settings to combinations of up to six skills.
- Compositional load: The formal load combines instance, emotion, text, and instance-weighted color, size, and spatial counts.
- Prompt generation: Synthetic prompts are built from meta-prompts, composed JSON skill rules, difficulty-based object assignments, and lexical or semantic perturbations.The resulting dataset contains 1,140 manually verified prompts.
- Model evaluation: The evaluation covers multiple proprietary and open-source models selected under resource, recency, documentation, and loading constraints.Generations used shared GPU infrastructure, while more than 5,500 images were generated for six models.
4 Experiments results
Across difficulty levels, T2I accuracy degrades non-uniformly as instance counts and combined skill constraints increase. Analyses controlling for prompt length indicate that compositional load explains more performance variation than surface linguistic complexity.
- Overall evaluation: WER measures exact lexical disagreement between generated and target text, with 0 indicating an exact match.Lower WER indicates higher accuracy.
- Overall evaluation: Gemini-Flash-3.1 led most metrics, but failed to generate coherent images in 32% of cases.It achieved the best overall performance on 4 of 7 metrics, excluding Counting.
- Difficulty and skill combinations: 243% higher error rates made Text the most degraded skill from easy to hard, followed by Counting, Cohesiveness, and Emotion.Their accuracy decreases were 71%, 60%, and 47%, respectively, while Color dropped 19%.
- Difficulty and skill combinations: Accuracy typically decreases as the number of combined skills increases, even when difficulty is fixed at the easy level.The same degradation appears across all difficulty levels.
- Prompt length versus compositional load: Prompt-length matching and multivariate regression separate compositional load from linguistic complexity when comparing difficulty levels.Compositional load has larger standardized coefficients and lower p-values than prompt length across structured skills, with limited multicollinearity indicated by VIF values below 5.
- TIIF-Bench analysis: More than 90% of analyzed TIIF-Bench prompt pairs gained compositional content beyond simple length expansion.The added content primarily involved atmosphere, texture, and contextual details that were formalized as additional rules.
- Typo robustness: Typo-conditioned prompts produced larger text deviations, and models introduced alternative spelling errors in 30% of cases.Figure 4b compares text rendering with and without injected typos.
5 Discussion
The discussion emphasizes that automated evaluation can streamline several skills but remains unreliable for interpretative cohesiveness judgments. It also identifies fixed skill sets as a boundary of current benchmark design.
- Human annotations and automation: Automated detection, depth estimation, segmentation, and vision-language inference support evaluation across counting, spatial, size, color, emotion, and cohesiveness skills.The pipeline uses YOLO-26 for several structured skills and Qwen3-VL for Color, Emotion, and Cohesiveness.
- Human annotations and automation: VQA alignment with human judgments rarely exceeded 70% for Cohesiveness despite attempts at instance-level evaluation and prompt reformulation.Traditional neural-network frameworks also failed to provide significant improvements, reinforcing the role of human oversight.
- Current evaluation practices: The benchmark relies on a fixed set of skills or predefined combinations, motivating future modular benchmarks that can add or remove skills as models evolve.The proposed direction would generate corresponding evaluation prompts on demand.
- Current evaluation practices: Table 3 reports stronger and more stable effects for compositional load than prompt length across most structured skills, while the relationship is less pronounced for Emotion and Cohesiveness.The table compares standardized effects of CL and L, with VIF included to assess multicollinearity.
6 Conclusion
Imag-Eval provides a controlled, interpretable framework and benchmark for diagnosing compositional instruction following in T2I models. Its findings indicate that structured-skill performance is driven more by compositional load than surface linguistic properties, while future work should support more modular evaluation.
- Imag-Eval disentangles surface linguistic complexity from compositional difficulty by varying compositional load: instance count and constraint combinations.
- The benchmark contains 1,140 prompts and more than 8,000 combined rules, with validation involving over 6,000 generated images and 14 annotators.
- Model performance for structured skills is primarily driven by grounded rules and their binding to instances rather than surface properties such as prompt length.
- The framework supports controlled variation of instance counts and prompt regeneration with fixed constraints.
- Future work should enable fully modular skill composition, allowing evaluation dimensions to be added, removed, or recombined dynamically.
Limitations
The evaluation is bounded by its COCO-based prompt vocabulary, single-seed image generation, single-prompt format, English-only prompts, and incompletely blinded annotation protocol. The framework is extensible and future versions are intended to address several of these boundaries.
- The framework currently uses prompts built from COCO object categories and attributes emotions only to human instances.
- All images were generated with random seed 42, so robustness across random initializations remains unmeasured in the present study.
- The method evaluates single-prompt instructions rather than multiprompt incremental sequences, whose rule-order effects lack a clear methodology.
- All prompts are in English, limiting direct generalization to languages with substantially different typological, morphological, or syntactic properties.
- Annotators were not fully blinded to model identity, which may have introduced bias in subjective judgments.
- Future benchmark versions should use fully anonymized filenames and randomized annotation interfaces.
- The JSON-based architecture can redefine difficulty levels and instance counts, while model-agnostic interfaces support proprietary and open-source systems.
C Dataset statistics
The dataset distributes rules across counting, color, spatial, emotion, size, and text categories, while cohesiveness is enforced globally rather than instantiated as an explicit rule. Annotation materials emphasize consistent treatment of interpretative cohesiveness failures and edge cases.
- The dataset contains 2,228 Counting, 1,660 Color, 2,199 Spatial, 372 Emotion, 2,197 Size, and 186 Text rules.
- Cohesiveness is enforced globally throughout evaluation instead of being instantiated as an explicit rule.
- Annotators received guidelines and examples for each skill, with cohesiveness requiring more interpretative judgment than criteria such as instance counting.
- The guidelines covered incomplete objects, anatomical inconsistencies, and other cohesiveness failure modes.
- Representative ambiguous generations and explicit handling instructions were documented for annotation.
- Figure 6 presents configuration parameters controlling difficulty granularity and the number of generated instances.
E Overall results for all the models
Overall results for all skill combinations are reported in Table 5, with one model excluded from the overall analysis because generation was incomplete under content-filtering and resource constraints.
- Results for all skill combinations are reported in Table 5.
- GEMINI-3.1-FLASH is excluded from the overall analysis because some images could not be generated due to content filtering and API budget or limit constraints.
F Overall results and impact of Compositional load
Across most structured skills, compositional load is more strongly associated with accuracy than prompt length. Performance declines as required skills increase, while Cohesiveness and Emotion are notable exceptions.
- Overall trend: Performance consistently degrades as the number of required skills increases, even when the number of instances remains fixed.This pattern extends across difficulty levels and most skill categories.
- Correlation analysis: Across most skills, compositional load correlates more strongly with skill-specific accuracy than prompt length.Cohesiveness and Emotion are the only exceptions, with marginal correlation differences.
- Regression analysis: Across most skills, compositional load has a strong, statistically significant negative regression effect, whereas prompt length has a small, non-significant effect.Standardized ordinary-least-squares coefficients support direct comparison of effect magnitudes; Cohesiveness and Emotion are exceptions.
- Regression analysis: Confidence intervals indicate that compositional load has a more consistent association with declining accuracy than prompt length.The regression analysis supports evaluating models along multiple complementary axes of difficulty.
- Overall conclusion: Taken together, the findings indicate that structured-skill performance is governed more by compositional load than by prompt length alone.This motivates controlled evaluation across multiple difficulty axes.
G Confusing cases
The paper identifies ambiguous generations that require manual validation, while documenting the released resources, generation settings, and aggregate model comparison for reproducibility and interpretation.
- Confusing cases: Some generations could not be reliably evaluated because instances were indeterminate or the model hallucinated excessively.These confusing cases motivated manual validation of all annotations.
- Resource release: The dataset, annotation guidelines, and associated research assets are intended for research, educational, and non-commercial R&D purposes.The resources are released under CC BY-NC 4.0, while source code uses the MIT License.
- Reproducibility: Generation parameters are listed for reproducibility, and the evaluated models were used consistently with their intended use.Table 4 identifies APL as Apache License.
- Aggregate comparison: Across 2- to 6-skill combinations, Z-Image-Turbo outperforms the other evaluated models on most metrics.Gemini was excluded because content-filter and budget issues caused missing images.