Source-linked AI summary
Davidsonian Scene Graph: Improving Reliability in Fine-grained Evaluation for Text-to-Image Generation
Jaemin Cho, Yushi Hu, Roopal Garg, Peter Anderson, Ranjay Krishna, Jason Baldridge, Mohit Bansal, Jordi Pont-Tuset, Su Wang
TL;DR
Text-to-image alignment evaluation needs fine-grained measures without unreliable question generation or inconsistent VQA answers. The paper introduces DSG, a dependency-aware representation that improves QG/A reliability, and releases DSG-1k with 1,060 diverse prompts for evaluation.
Problem
Existing QG/A evaluation can generate ambiguous, duplicated, hallucinated, or omitted prompt content, while VQA can produce inconsistent answers for dependent questions.
Method
DSG decomposes prompts into atomic, unique questions organized by semantic dependency graphs, skipping dependent questions after negative parent answers.
Results
DSG better addresses QG/A reliability issues than previous work, while experiments and human evaluation assess LLM, VQA, and T2I configurations.
Takeaways & Limitations
DSG provides fine-grained diagnostic evaluation and DSG-1k supplies 1,060 prompts spanning diverse semantic categories and writing styles.
Takeaways & Limitations
PaLM2 and PaLI were unreleased, and human evaluation may not be exactly reproducible with different raters.
Abstract
from arXiv · showhide
Evaluating text-to-image models is notoriously difficult. A strong recent approach for assessing text-image faithfulness is based on QG/A (question generation and answering), which uses pre-trained foundational models to automatically generate a set of questions and answers from the prompt, and output images are scored based on whether these answers extracted with a visual question answering model are consistent with the prompt-based answers. This kind of evaluation is naturally dependent on the quality of the underlying QG and VQA models. We identify and address several reliability challenges in existing QG/A work: (a) QG questions should respect the prompt (avoiding hallucinations, duplications, and omissions) and (b) VQA answers should be consistent (not asserting that there is no motorcycle in an image while also claiming the motorcycle is blue). We address these issues with Davidsonian Scene Graph (DSG), an empirically grounded evaluation framework inspired by formal semantics, which is adaptable to any QG/A frameworks. DSG produces atomic and unique questions organized in dependency graphs, which (i) ensure appropriate semantic coverage and (ii) sidestep inconsistent answers. With extensive experimentation and human evaluation on a range of model configurations (LLM, VQA, and T2I), we empirically demonstrate that DSG addresses the challenges noted above. Finally, we present DSG-1k, an open-sourced evaluation benchmark that includes 1,060 prompts, covering a wide range of fine-grained semantic categories with a balanced distribution. We release the DSG-1k prompts and the corresponding DSG questions.
1 INTRODUCTION
QG/A enables fine-grained, interpretable T2I alignment evaluation, but existing methods suffer from ambiguous, duplicated, hallucinated, and dependency-invalid questions. DSG addresses these reliability issues with atomic, unique questions organized by semantic dependencies, and experiments show improved reliability and diagnostic value.
- Motivation: QG/A generates validation questions and expected answers from prompts, then uses VQA on generated images to compute alignment scores.This provides more fine-grained and interpretable evaluation than single-summary similarity metrics.
- Motivation: 47.2 Kendall’s τ versus 23.1 for CLIPScore demonstrates the reported human-rating correlation advantage of QG/A.The comparison comes from a sizable correlation analysis using human 1–5 Likert judgments.
- Reliability Challenges: Existing QG/A methods produce ambiguous, duplicated, hallucinated, and invalid questions that can make evaluation unreliable.Examples include combining object presence with color and asking dependent questions after a negative existence answer.
- DSG: DSG represents prompt semantics with atomic propositions, unique questions, and dependency edges in a directed acyclic graph.Negative answers cause dependent questions in the corresponding subgraph to be skipped.
- Results: Experiments report that DSG addresses the identified reliability issues, while VQA models remain capable in some semantic categories but fail in others.The framework is also used for fine-grained diagnosis of state-of-the-art T2I models.
2 RELATED WORK
Related work contrasts coarse single-summary T2I metrics with QG/A evaluation, which uses generated questions and VQA answers to assess prompt-image alignment more granularly.
- Single-summary scoring frameworks: Common T2I metrics use text-image embedding similarity or image-caption text similarity, but single embeddings obscure semantic granularity.The related-work discussion also notes that such metrics can be uncalibrated across image types.
- QG/A frameworks: QG/A evaluates multimodal alignment by applying VQA to questions generated from the prompt with pretrained foundation models.This approach extends question-answering-based evaluation from text summarization into the multimodal domain.
- QG/A frameworks: QG/A frameworks provide a fine-grained alternative to single-summary scoring for diagnosing text-to-image alignment.The supplied related-work passages frame this as the central transition in evaluation methodology.
3 DAVIDSONIAN SCENE GRAPH
DSG models prompt semantics as atomic propositions in a dependency graph, translates them into VQA questions, and aggregates answer agreement into alignment scores. Its automated pipeline and DSG-1k benchmark support structured, category-level T2I evaluation.
- DSG evaluation: DSG evaluates a generated image by comparing VQA answers with prompt-derived expected answers and aggregating per-question accuracy.Closer agreement between predicted and expected answers indicates greater prompt-image alignment.
- Scene-graph representation: Atomic propositions form a DAG whose edges encode entailment dependencies, so child questions are evaluated only when their parent propositions hold.For example, motorcycle color is evaluated only if motorcycle existence is established.
- Scene-graph representation: DSG represents entities and globals as 1-tuples, attributes as 2-tuples, and relationships as 3-tuples, each translated into an atomic question.The resulting directed acyclic graph is the Davidsonian Scene Graph.
- Automatic generation: Automatic DSG generation uses three stages: prompt-to-tuples, tuples-to-questions, and tuples-to-dependencies.Each tuple receives an identifier, questions have expected binary yes/no answers, and parent-child dependencies are generated from identifier pairs.
- DSG-1k: DSG-1k contains 1,060 prompts sampled from multiple datasets to cover diverse skills, semantic categories, and writing styles.The benchmark expands beyond TIFA160 to increase prompt diversity.
4 EXPERIMENTS AND DISCUSSION
The experiments evaluate DSG question generation and VQA-based T2I assessment across semantic matching, atomicity, uniqueness, dependency validity, human correlation, and answer agreement. DSG achieves high semantic coverage and reliable question structure, while VQA performance remains uneven across semantic categories.
- Question Generation: DSG questions match human-annotated semantic tuples with 92.2% precision and 100.0% recall manually, and 98.3% precision and 96.0% recall automatically.The manual evaluation used 30 prompts, while the GPT-3.5 evaluation covered all 160 TIFA160 prompts.
- Question Generation: DSG achieves 90.5% atomicity, exceeding TIFA at 73.4% and VQ2A at 79.8%, while reducing ambiguity from multi-detail questions.Atomic questions isolate one entity, attribute, or relation, making answers easier to interpret.
- Question Generation: DSG dependency validity reaches 100% on 30 manually evaluated TIFA160 samples and 99% on the full dataset, barring a few parsing errors.Child questions are posed only when their parent question receives a positive answer, avoiding invalid VQA queries.
- Question Answering: DSG+PaLI obtains the strongest per-item human correlation, with Spearman’s ρ = 0.563 and Kendall’s τ = 0.458.These correlations compare VQA-based scores with human 1-5 Likert ratings on TIFA160 text-image pairs.
- Question Answering: PaLI has the highest per-question VQA-human match ratio at 73.8%, with entity questions easier than attribute, relation, and global questions.Visually explicit attributes such as color and texture are easier than counting, text rendering, and other higher-order categories.
- Text-to-Image Evaluation: PaLI ranks Imagen* and MUSE* above SD v2.1 like human ratings, but its score scale is overly optimistic at roughly 80% versus roughly 60% human accuracy.The largest discrepancies occur for counting, real-user prompts, and text, where VQA or T2I systems often struggle.
5 CONCLUSION
The paper proposes DSG to address reliability problems in QG/A-based text-to-image evaluation and introduces DSG-1k to support further research.
- DSG addresses duplicated, invalid, and ambiguous questions in QG/A evaluation through a formal-semantics-inspired approach.
- Comprehensive experiments and human evaluation show that DSG better addresses these reliability issues than previous work.
- DSG-1k is a benchmark dataset collected to facilitate research in fine-grained text-to-image evaluation.
- The authors identify subjectivity, domain knowledge, and difficult semantic categories such as text rendering as limits for QG and current VQA models.
6 STATEMENTS
The statements describe human-evaluation compensation and reproducibility conditions, including unavailable models and possible variation in human-evaluation results.
- Human raters were paid contractors receiving standard wages compliant with living-wage laws in their countries of employment.
- Experiments using publicly available packages and tools are reproducible using details provided in the main text and appendix.
- The reproducible experiments include GPT-3.5 question evaluation, VQA results with mPLUG-large and InstructBLIP, and Stable Diffusion images.
- PaLM2 and PaLI are not yet released, and human-evaluation results may vary with different raters.
- The appendix documents DSG-1k details, invalid VQA examples, ablations, additional evaluations, mixed-model experiments, pseudocode, and human-evaluation setup.
A LLM PREAMBLES
The appendices describe preambles for generating DSG questions and for automatically evaluating their precision, recall, and uniqueness.
- LLM Preambles: Figure 7 presents preambles for the three-stage automatic DSG generation pipeline using PaLM2.
- Question Evaluation: Figure 8 presents preambles for evaluating question precision, recall, and uniqueness with GPT-3.5.
- Question Evaluation: The precision task checks which generated questions are entailed by ground-truth prompt tuples and identifies questions that are not entailed.
B DSG-1K DATASET DETAILS
DSG-1k combines prompts from multiple sources with generated images and question annotations, while documenting question statistics and invalid VQA examples.
- Dataset and Images: 3,180 images were generated from 1,060 DSG-1k prompts using Imagen, MUSE, and Stable Diffusion v2.1.
- Questions and Evaluation: DSG questions and dependencies were generated for DSG-1k, then shown with images to human annotators for validity judgments and alignment evaluation.
- Dataset Statistics: DSG-1k statistics support the validity of category analyses by showing sufficient question counts across categories.
- Dataset Statistics: The dataset includes statistics for broad and detailed semantic categories, including entity, relation, and attribute categories.
- Invalid VQA Queries: Invalid VQA queries can occur when a dependent question receives a positive answer even though its root question receives a negative answer.
- Invalid VQA Queries: Such invalid queries are more common when T2I models miss entities, particularly in weaker models such as minDALL-E and VQ-Diffusion.
D.1 WAYS TO UTILIZE DEPENDENCY
DSG supports three dependency-handling strategies for invalid child questions. Treating parent dependencies as part of per-question scoring substantially improves VQA–human matching, while aggregate correlations are largely unchanged.
- Dependency strategies: DSG propagates a parent NO answer to all child questions, whereas DSG (drop) excludes those children and DSG (w/o dep) scores all questions equally.These variants differ in how invalid dependent questions contribute to VQA accuracy.
- Aggregate correlation: Dependencies do not significantly affect correlation between aggregate VQA accuracy and human 1–5 Likert judgments.This result concerns the aggregate correlation analysis.
- Per-question matching: Using dependencies with either DSG or DSG (drop) substantially improves per-question VQA–human matching accuracy.The improvement is reported for per-question matching rather than aggregate correlation.
- Interpretation: The authors interpret the per-question result as evidence that human raters implicitly account for question dependencies.They argue this supports dependencies for more reliable and interpretable fine-grained evaluation.
- Data-split analysis: The TIFA160 ablation uses COCO, PaintSkill, PartiPrompts, and DrawBench splits covering general purpose, spatial, attributional, and compositional prompts.The split analysis extends the correlation comparison across these four prompt sources.
E ADDITIONAL QG/QA/T2I EVALUATION RESULTS
Additional evaluations compare DSG with single-summary metrics and report question-level and model-level results across DSG-1k and TIFA160. DSG achieves higher correlations with human judgments across the evaluated VQA modules, while the analyses also expose category-specific variation.
- Caveat: Human scores used for the single-summary comparison came from different raters than the prior reported results, limiting direct applicability of that table.The paper explicitly cautions that the earlier table cannot be directly applied.
- Single-summary comparison: DSG produces much higher correlations with human scores than the investigated single-summary metrics across all VQA modules.The comparison includes BLEU-4, ROUGE-L, METEOR, SPICE, and CLIPScore.
- Single-summary comparison: The single-summary comparison reports Spearman and Kendall correlations on TIFA160 prompts across the same five models used by prior work.The metrics include captioning-based scores and CLIPScore.
- QA evaluation: DSG-1k QA evaluation reports VQA–human answer-match accuracy by data source, with trends matching the by-category results.The results use DSG-PaLM2 questions on three T2I generation models.
- T2I evaluation: TIFA160 provides average 1–5 Likert scores for the evaluated T2I generation models.The additional category analysis reports VQA–human correlations and accuracy for categories with at least 30 samples and p < 1e-4.
F EXPERIMENTS ON MIXED QA EVALUATION MODELS
The mixed-model experiments test domain-specific object detection and OCR alongside direct VQA within the DSG evaluation pipeline. OCR yields a noticeable but not categorically substantial improvement in VQA–human alignment, whereas object detection performs slightly worse.
- Mixed QA models: Domain-specific models produce different VQA–human alignment results: object detection is slightly lower, while OCR gives a noticeable but not categorically substantial bump.These findings are consistent with the cited concurrent-work results.
- Category analyses: The evaluation reports VQA–human correlations and T2I model VQA accuracy by fine-grained entity, relation, and attribute categories.The category analyses distinguish VQA correctness from human 1–5 Likert judgments.
- Object detection: Object detection is evaluated for counting on CountBench by comparing direct VQA with OWLv2-base-patch16 queries.The comparison spans TIFA, VQ2A, and PaLI configurations.
- OCR: OCR is evaluated for text rendering on DrawText by comparing direct VQA with EasyOCR-integrated VPEval queries.The corresponding results are summarized as direct-VQA versus OCR-assisted matching accuracy.
- Evaluation pipeline: Algorithm 1 implements DSG by generating tuples, dependencies, and questions, answering them with VQA, zeroing invalid dependent-question scores, and averaging the results.The pseudocode exposes the pipeline stages but not a separate mixed-model outcome.
H HUMAN EVALUATION SETUP
The human evaluation setup includes interfaces for collecting judgments at both the question and text-image-item levels. Item-level judgments use a 1–5 Likert consistency rating.
- Question-level annotation: The study uses a per-question interface for collecting human judgments.This interface supports question-level annotation.
- Item-level annotation: The study uses a per-item interface with a 1–5 Likert consistency rating.This interface collects text-image pair judgments at the item level.