PaperlayerView exact revision on arXiv

Source-linked AI summary

T2I-CompBench++: An Enhanced and Comprehensive Benchmark for Compositional Text-to-image Generation

Kaiyi Huang, Chengqi Duan, Kaiyue Sun, Enze Xie, Zhenguo Li, Xihui Liu

arXiv:2307.06350v3cs.CV

TL;DR

Text-to-image models still struggle to compose multiple objects with correct attributes and relationships, while existing benchmarks and metrics provide incomplete coverage. T2I-CompBench++ introduces an 8,000-prompt benchmark, tailored metrics, MLLM analysis, and GORS; experiments validate the metrics, show consistent GORS gains, and benchmark 11 models. The study also identifies limits in current metrics and MLLM-based evaluation.

  • Problem

    Text-to-image models often fail to compose complex scenes correctly, and existing benchmarks and similarity metrics inadequately evaluate compositionality.

  • Method

    The paper builds T2I-CompBench++ with 8,000 prompts across four categories and eight sub-categories, adds category-specific metrics and MLLM analysis, and proposes reward-driven GORS fine-tuning.

  • Results

    Experiments across four categories and eight sub-categories validate the evaluation metrics, while GORS consistently outperforms baseline models and 11 models are benchmarked.

  • Takeaways & Limitations

    The benchmark provides a comprehensive framework for assessing compositional text-to-image generation and a baseline for improving compositional abilities.

  • Takeaways & Limitations

    The work lacks a unified metric for all forms of compositionality, and its metrics have documented failure cases and scope limitations.

Abstract

from arXiv · show

Despite the impressive advances in text-to-image models, they often struggle to effectively compose complex scenes with multiple objects, displaying various attributes and relationships. To address this challenge, we present T2I-CompBench++, an enhanced benchmark for compositional text-to-image generation. T2I-CompBench++ comprises 8,000 compositional text prompts categorized into four primary groups: attribute binding, object relationships, generative numeracy, and complex compositions. These are further divided into eight sub-categories, including newly introduced ones like 3D-spatial relationships and numeracy. In addition to the benchmark, we propose enhanced evaluation metrics designed to assess these diverse compositional challenges. These include a detection-based metric tailored for evaluating 3D-spatial relationships and numeracy, and an analysis leveraging Multimodal Large Language Models (MLLMs), i.e. GPT-4V, ShareGPT4v as evaluation metrics. Our experiments benchmark 11 text-to-image models, including state-of-the-art models, such as FLUX.1, SD3, DALLE-3, Pixart-$α$, and SD-XL on T2I-CompBench++. We also conduct comprehensive evaluations to validate the effectiveness of our metrics and explore the potential and limitations of MLLMs.

I. INTRODUCTION

T2I-CompBench++ addresses inconsistent definitions and inadequate compositionality metrics by expanding benchmark coverage and introducing category-specific evaluation. It also benchmarks 11 text-to-image models and proposes GORS for improving compositional generation.

  • State-of-the-art text-to-image models often misbind attributes to objects or generate incorrect spatial layouts in complex scenes.
  • Existing compositional benchmarks target narrow sub-problems, while CLIPScore and BLIP-based evaluations struggle with compositional vision-language understanding.
  • T2I-CompBench++ covers four categories and eight sub-categories, adding generative numeracy and 3D-spatial relationships.
  • The benchmark introduces Disentangled BLIP-VQA, UniDet-based, and MLLM-based metrics tailored to different compositional categories.
  • GORS fine-tunes Stable Diffusion v2 using reward-weighted losses on generated images aligned with compositional prompts.
  • Experiments validate the metrics through human correlation, report consistent GORS gains over baselines, and benchmark 11 models including FLUX.1, SD3, DALLE-3, Pixart-α, and SD-XL.

II. RELATED WORK

Prior work studies compositional generation through isolated capabilities, constrained benchmarks, and varied evaluation metrics. T2I-CompBench++ frames compositionality comprehensively and constructs a balanced, scalable prompt benchmark spanning diverse compositions.

  • Text-to-image research progressed from GAN-based systems to diffusion models trained on web-scale data.
  • Earlier compositional studies separately address concept conjunction, attribute binding, numeracy, or spatial relationships, often in constrained scenarios.
  • Existing benchmarks vary from narrow attribute-binding datasets to broad skill suites where compositionality receives limited emphasis.
  • T2I-CompBench++ defines compositionality across attribute binding, object relationships, numeracy, and complex compositions, with eight sub-categories and 8,000 prompts.
  • The benchmark balances seen and unseen compositions, fixed-template and natural prompts, and simple and complex prompts.
  • Prompt construction combines predefined rules with ChatGPT generation, including intentionally physically implausible scenarios to test generalization.

B. Object Relationship

The benchmark evaluates object relationships across 2D spatial, 3D spatial, and non-spatial interactions using structured prompt categories and varied object types.

  • Object relationship categories: 1,000 prompts cover 2D/3D-spatial and non-spatial relationships, respectively.
  • 2D-spatial relationships: 2D relationships include relative positions and proximity terms such as left, right, top, bottom, next to, and near.
  • 2D-spatial relationships: Contrastive 2D prompts swap the two nouns to test whether models preserve directional relationships.
  • 3D-spatial relationships: 3D relationships are defined with “in front of,” “behind,” and “hidden by” using paired object prompts.
  • Non-spatial relationships: Non-spatial prompts describe interactions such as wearing, holding, watching, speaking, playing, walking, and sitting.

C. Numeracy.

The numeracy category tests object counting in increasingly complex prompts, while the evaluation combines detection, spatial reasoning, and multimodal metrics for compositional assessment.

  • Numeracy dataset: 1,000 numeracy prompts allocate 30% to one object, 30% to two objects, and 40% to multiple objects with quantities from one to eight.
  • Complex compositions: Complex-composition prompts use more than two objects and multiple attributes, extending beyond predefined prompt patterns.
  • Evaluation motivation: Existing CLIPScore and BLIP-CLIP metrics capture coarse similarity but miss fine-grained attribute binding and spatial correspondences.
  • Evaluation motivation: BLIP captioning may omit detailed object attributes, making direct text-text comparison ambiguous for attribute binding.
  • Attribute binding evaluation: Disentangled BLIP-VQA asks independent object-attribute questions and multiplies their “yes” probabilities into an overall score.
  • Detection-based evaluation: Detection-based evaluation is proposed for spatial relationships and numeracy because vision-language models struggle with spatial reasoning and counting.
  • Detection-based evaluation: UniDet evaluates 2D relationships from bounding-box centers and distances, using an IoU threshold below 0.1 for directional relations.
  • Detection-based evaluation: 3D evaluation combines object detection with depth estimation; d1 > d2 and IoU above 0.5 indicate the first object is in front.

C. 3-in-1 Metric for Complex Compositions Evaluation

Because different metrics suit different compositional categories, the paper combines specialized non-MLLM metrics and examines MLLMs as a broader evaluation option.

  • 3-in-1 metric: No single metric performs well across all compositionality categories.
  • 3-in-1 metric: The proposed 3-in-1 metric averages category-specific scores selected for attribute binding, spatial relationships, numeracy, and non-spatial relationships.
  • MLLM evaluation: MLLM evaluation submits generated images to a model and measures alignment with the supplied text prompt.
  • MLLM evaluation: The evaluation uses Chain-of-Thought prompts that first request an image description and then an image-text alignment score.
  • MLLM evaluation: Non-Chain-of-Thought prompts ask MLLMs to score object-attribute presence and correctness from 0 to 100.

V. BOOSTING COMPOSITIONAL TEXT-TO-IMAGE GENERATION WITH GORS

GORS improves pretrained text-to-image models by selecting highly aligned generated samples and weighting fine-tuning by their compositional rewards, within a broad multi-model evaluation setup.

  • GORS: GORS fine-tunes pretrained text-to-image models with generated images that align strongly with compositional prompts.
  • GORS: The method weights the fine-tuning loss by each selected sample’s image-text alignment reward.
  • GORS procedure: For each prompt, GORS generates k images, predicts rewards, retains samples above a threshold, and fine-tunes on the selected set.
  • GORS procedure: Higher-alignment images receive higher loss weights during fine-tuning.
  • Implementation: GORS uses LoRA for efficient fine-tuning, including attention components in the CLIP text encoder and U-Net.
  • Experimental setup: The experiments evaluate 11 text-to-image models, including five state-of-the-art systems and the proposed GORS approach.
  • Experimental setup: GORS-unbiased uses Grounded-SAM rewards to reduce bias from selecting samples with the proposed evaluation metrics.

B. Evaluation Metrics

The paper evaluates compositional generation with category-specific automatic metrics, human evaluation, and MLLM-based scoring. Proposed metrics better track human judgments than prior similarity-based metrics.

  • 10 images per prompt are generated for automatic evaluation using a fixed seed across models.
  • CLIP, BLIP-CLIP, and BLIP-VQA-naive are used as previous evaluation metrics.
  • Disentangled BLIP-VQA evaluates attribute binding, while UniDet evaluates 2D/3D spatial relationships and numeracy.
  • MiniGPT4, ShareGPT4V, and GPT-4V are evaluated as MLLM metrics for non-spatial relationships and complex compositions.
  • The proposed metrics show trends similar to human evaluation, whereas CLIP and BLIP-CLIP predict similar scores across models.
  • Human-correlation results support BLIP-VQA for attribute binding and UniDet-based metrics for spatial relationships and numeracy.

D. Discussion about MLLMs as a unified metric

The discussion compares MLLMs as unified evaluation metrics across compositional categories and examines prompting, stability, and limitations. MLLM performance varies by category, and Chain-of-Thought generally improves correlation for tested models.

  • ShareGPT4V excels in non-spatial and complex categories but performs slightly below non-MLLM metrics in color, texture, spatial, and numeracy categories.
  • mGPT and ShareGPT4V are treated as unified entities by comparing their highest human-correlation values across prompting variants.
  • GPT-4V ranks first in non-spatial relationships and complex compositions, while its ranks vary across attribute, spatial, and numeracy categories.
  • Chain-of-Thought improves MiniGPT4 and ShareGPT4V across most categories in human correlation.
  • Without Chain-of-Thought, MiniGPT-4 evaluation does not strictly align with human evaluation results.
  • Repeated evaluations vary within 0.032 for GPT-4V and 0.0273 for ShareGPT4V, with average standard deviations of 7.3235 and 6.8741, respectively.
  • Share-CoT gives less diverse ratings, while GPT-4V understands grading prompts but is less able to convert them into exact grades.

E. Benchmarking on Different Methods

The benchmark compares text-to-image methods across compositional categories, splits, training-set scales, and prompt formulations. Performance differs substantially by model and task, with spatial relationships remaining especially difficult.

  • Stable Diffusion v2 consistently outperforms Stable Diffusion v1-4 across compositional prompts and evaluation metrics.
  • DALLE 3 and SD3 achieve state-of-the-art performance by a significant margin in almost all categories.
  • Spatial relationships are the most challenging sub-category, shape attribute binding is also challenging, and non-spatial relationships are easiest.
  • The unseen attribute-binding split contains more uncommon attribute-object combinations and yields slightly lower performance than the seen split.
  • Scaling complex-composition training data from 25 to 1,400 prompts increases the model’s performance.
  • More detailed rephrased prompts do not significantly improve generative outcomes, although they can improve image quality through additional descriptions.

F. Ablation Study

The ablations study finetuning components, sample selection, and metric thresholds for the proposed approach. Joint finetuning and selecting well-aligned samples improve compositional behavior, while a threshold of 0.5 best correlates with human evaluation.

  • Finetuning strategy: Jointly finetuning the CLIP text encoder and U-Net with LoRA performs better than finetuning either component alone.
  • Sample selection: A higher sample-selection threshold includes more text-aligned images, whereas a lower threshold admits misaligned images that can degrade compositional ability.
  • Metric threshold: 0.5 achieves the highest human correlation among tested UniDet thresholds of 0.0, 0.25, 0.50, and 0.75.

G. Limitation and Potential Negative Social Impacts

The paper acknowledges limitations in its evaluation metrics and warns about social risks and biases associated with generative and multimodal models.

  • The work lacks a unified metric covering all forms of compositionality.Future research could explore multimodal LLMs for unified evaluation.
  • BLIP-VQA can fail when object shapes are obscured, descriptions are uncommon, or objects are difficult to recognize.
  • The UniDet-based metric is limited to evaluating 2D spatial relationships, leaving 3D spatial relationships for future study.
  • The authors warn that text-to-image models and pretrained multimodal evaluators may be abused and may exhibit hallucination-related biases.

APPENDIX

Appendix evaluations show that added prompt detail does not resolve compositional difficulties, while GORS preserves long-prompt handling and UniDet thresholds distinguish qualitative cases. Even state-of-the-art models struggle with complex compositional prompts.

  • 24.00 versus 8.21 words did not significantly improve outcomes after GPT-4 rephrased prompts with greater detail.The appendix reports that compositional challenges persist across the original and rephrased prompt sets.
  • GORS preserves the ability to handle long text prompts rather than becoming overly attuned to short prompts.The comparison uses Stable v2 as the base model and Stable v2+GORS as the enhanced model.
  • The chosen UniDet threshold effectively distinguishes good and bad cases for 2D/3D spatial relationships and generative numeracy.
  • Even state-of-the-art models struggle with complex prompts involving non-spatial relationships and at least three objects.The qualitative examples cover FLUX.1, SD3, and DALL-E 3.
  • The appendix contrasts original prompts such as “A blue bench and a green bowl” with more detailed descriptions of the same scene.
Loading 2307.06350v3…