Davidsonian Scene Graph: Improving Reliability in Fine-grained Evaluation for Text-to-Image Generation

1 INTRODUCTION

• QG/A generates validation questions and expected answers from a prompt, then scores alignment by comparing image-based VQA responses with those expected answers.This provides a fine-grained alternative to evaluating prompt–image similarity using captioning models, multimodal encoders, or object detectors.Source: §1 · p. 1
• 47.2 Kendall’s τ is reported for Hu et al. (2023) against human 1–5 Likert judgments, compared with CLIPScore τ = 23.1.QG/A also offers better calibration and more granular interpretation than single-score similarity metrics, distinguishing errors in entities, spatial relations, and colors.Source: §1 · pp. 1–2
• Existing QG/A methods generate ambiguous multi-detail questions, duplicates, hallucinations, and contradictory answers to dependent questions.Duplicate questions overrepresent particular prompt content and skew the final metric; dependent questions can assert an absent object’s attributes.Source: §1 · p. 2
• DSG represents prompts as unique, atomic questions in a directed acyclic graph, skipping all dependent questions after a negative answer.Questions cover all and only the prompt’s semantics, with atomic units enabling unambiguous interpretation. The modular pipeline combines PaLM 2 340B with PaLI 17B (Chen et al., 2023).Source: §1 · p. 2
• 1,060 prompts comprise DSG-1k, a human-annotated benchmark with balanced semantic categories and styles, sourced from diverse public datasets.The benchmark is intended to facilitate research in fine-grained T2I alignment evaluation.Source: §1 · p. 2
• Experiments find that DSG addresses the identified reliability issues and supports fine-grained T2I diagnosis, while VQA models succeed in some semantic categories but fail in others.The paper evaluates state-of-the-art T2I models to demonstrate DSG’s diagnostic capabilities.Source: §1 · pp. 2–3

Single-summary scoring frameworks

• Embedding-similarity and caption-based scores are often uncalibrated, while single embeddings obscure the semantic granularity of text–image alignment.Examples include CLIPScore (Hessel et al., 2022), R-precision (Xu et al., 2018), and caption-based text similarity (Hong et al., 2018).Source: §2 · p. 3
• SOA and DALL-Eval use object detection for finer-grained evaluation but have limited coverage of relations and global style.Object-alignment accuracy provides calibration, but relations and global stylistic features often exceed object detection capabilities.Source: §2 · p. 3

QG/A frameworks

• DSG follows QG/A but draws on formal semantics (Davidson, 1965) to address reliability issues such as duplicated and non-atomic questions.Source: §2 · p. 3
• QG/A evaluates fine-grained alignment by answering prompt-generated visual questions about object counts, relations, and attribute bindings.TIFA (Hu et al., 2023) generates semantic-category questions with GPT-3 (Brown et al., 2020) and uses VQA modules such as mPLUG (Li et al., 2022). Yarom et al. (2023) use VQ2A (Changpinyo et al., 2022), improving VQA–human correlation through data synthesis and high-quality negative sampling.Source: §2 · p. 3
• VPEval (Cho et al., 2023b) adds object detection and optical character recognition, orchestrating queries through controllable visual programming with ChatGPT.Source: §2 · p. 3

Dependency-aware atomic decomposition of scene description

• DSG structures atomic propositions into an entailment-dependency graph, requiring parent propositions to hold before their dependent propositions can be evaluated.Questions derived from the prompt are answered by a pretrained VQA model against the generated image; comparison with expected answers yields per-question results and aggregated VQA accuracy.Source: §3 · pp. 3–4
• DSG encodes entities and globals as single-element tuples, attributes as two-element tuples, and relationships as three-element tuples, translating each into an atomic question.Attribute tuples pair an attribute with its entity; relationship tuples contain the relationship, subject entity, and object entity.Source: §3 · p. 4

Automatic generation of DSGs

• Automatic DSG generation uses three LLM stages: extracting numbered semantic tuples, translating tuples into binary questions, and identifying parent–child dependencies.Each stage uses task-specific in-context examples. The three-stage approach proved more effective than generating the entire DSG at once, allowing more than 20 examples per stage versus fewer than 10.Source: §3 · p. 4, Figure 4 · p. 4
• Two quality-assurance layers target atomicity, full semantic coverage, uniqueness, and valid dependencies: expert consensus on in-context examples and automatic plus human evaluation of generated DSGs.All three human experts must agree that each example question or question set satisfies the required properties.Source: §3 · pp. 4–5

DSG-1k dataset

• 1,060 prompts constitute DSG-1k, combining TIFA160 with 900 additional prompts from nine datasets to broaden skills, semantic categories, and writing styles.TIFA160, from TIFA v1.0 (Hu et al., 2023), supplies human Likert-scale alignment ratings. The expanded benchmark covers challenges including counting, text rendering, and relations, enabling source-specific scores.Source: §3 · p. 5

4.1 QUESTION GENERATION

• 92.2% precision and 100% recall manually, and 98.3% precision and 96.0% recall automatically, show DSG questions closely match source semantic tuples.Manual evaluation covers 30 TIFA160 prompts, while automated evaluation covers all 160. Remaining mismatches concern relation decomposition, referential scope, and subjective details that evaluation should exclude when not visually verifiable.Source: §4.1 · p. 6, Table 2
• 90.5% manual uniqueness for DSG exceeds VQ2A’s 79.8% and TIFA’s 73.4%, while DSG also improves question atomicity.Atomic questions ask about only one fact, reducing answer ambiguity, while uniqueness avoids skewing metric calculation. VQ2A atomicity is only 3.9%.Source: §4.1 · p. 6, Table 2
• 100% dependency validity on 30 manually evaluated prompts and 99% automatically across TIFA160 support DSG’s parent–child question structure.A child question is valid only when its parent receives a positive answer, helping avoid invalid VQA queries. The full-set evaluation has a few parsing failures.Source: §4.1 · p. 6

4.2 QUESTION ANSWERING

• 0.563 Spearman’s ρ and 0.458 Kendall’s τ make DSG+PaLI the best reported combination for correlation with human text–image alignment ratings.Evaluation compares average VQA scores against human 1–5 Likert ratings on TIFA160; DSG questions achieve the highest correlations among the question-generation methods.Source: Table 3 · p. 7, §4.2 · p. 7
• 73.8% VQA–human answer matching makes PaLI the strongest tested VQA model, slightly ahead of mPLUG-large and Instruct-BLIP.The DSG-1k evaluation uses DSG-PaLM2 questions and images from three generation models. All three VQA models answer entity questions more accurately than attribute, relation, or global questions.Source: §4.2 · p. 7, Table 4
• Below 40% model–human matching on text rendering exposes weaknesses in higher-order semantics despite stronger performance on visually explicit attributes.Color and texture are easier than counting and text rendering. The authors attribute difficulties with material, shape, and style partly to subjectivity and domain knowledge, which also undermine human judgments.Source: §4.2 · pp. 7–8, Table 5 · p. 8

4.3 TEXT-TO-IMAGE EVALUATION

• Approximately 80% accuracy for DSG+PaLI versus approximately 60% for humans accompanies matching model rankings: Imagen* ≃ MUSE* > SD v2.1.The comparison uses DSG-PaLM2 questions on DSG-1k. Imagen* and MUSE* are differently parameterized variants trained on internal data sources.Source: §4.3 · p. 8, Table 6 · p. 9
• Approximately 40% counting errors for PaLI illustrate why useful concrete-category model rankings do not establish reliable evaluation of difficult semantic details.PaLI tracks humans closely on TIFA160, Paragraph, Relation, and Commonsense-defying prompts, but overestimates Counting, Real users, and Text scores. Text failures, subjective descriptions, and knowledge-dependent judgments limit these categories.Source: §4.3 · pp. 8–9

5 CONCLUSION

• Comprehensive experimentation and human evaluation show that DSG addresses QG/A reliability issues better than previous work.Inspired by formal semantics (Davidson, 1965; 1967a;b), DSG targets duplicated, invalid, and ambiguous questions.Source: §5 · p. 9
• DSG’s fine-grained question typing enables more in-depth diagnostics for text-to-image evaluation.The authors also collect the DSG-1k benchmark dataset to facilitate further research.Source: §5 · p. 9
• Subjectivity and domain knowledge can cause model-human and human-human disagreement in both question generation and answering.These disagreements remain challenges for future QG/A research.Source: §5 · p. 9
• Some semantic categories, including text rendering, remain beyond the capabilities of existing state-of-the-art VQA models.Source: §5 · p. 9

6.1 ETHICS STATEMENT

• Human raters were paid contractors recruited through Google’s crowdsourcing platform, receiving standard contracted wages compliant with living-wage laws in their countries of employment.Raters examined images and text prompts and answered questions about text-image grounding.Source: §6.1 · p. 10
• Global privacy concerns prevent disclosure of further participant details, including estimated hourly wages and total compensation expenditure.Source: §6.1 · p. 10

6.2 REPRODUCIBILITY STATEMENT

• Experiments using publicly available packages and tools are reproducible, with necessary details provided in the main text and appendix.These include GPT-3.5 precision, recall, and uniqueness evaluation, plus VQA queries using mPLUG-large and InstructBLIP (Dai et al., 2023). They also include Stable Diffusion images used throughout Sec. 4.3.Source: §6.2 · p. 10
• PaLM2 and PaLI (Chen et al., 2023), used in the experiments, were not yet released.Their unreleased status limits reproduction of experiments relying on these models.Source: §6.2 · p. 10
• Human evaluation results may not reproduce exactly with different raters, although the authors expect general trends and conclusions to remain unchanged.Source: §6.2 · p. 10

Appendix

• The appendix supplements the paper with methodological materials, additional evaluations, ablations, mixed-model experiments, and implementation details.Appendices A–D cover LLM preambles, DSG-1k details, invalid VQA query examples, and DSG ablations. Appendices E–H cover additional QG/QA/T2I results, mixed QA evaluation models, DSG-based T2I evaluation pseudocode, and human-evaluation setup.Source: Appendix · p. 13

• Figure 7 provides preambles for the three-stage automatic DSG generation pipeline using PaLM2.Source: A LLM PREAMBLES · p. 14, Figure 7 · p. 14
• Figure 8 provides preambles for evaluating question precision and recall using GPT-3.5.Source: A LLM PREAMBLES · p. 14, Figure 8 · p. 14
• The question-evaluation preambles in Figure 8 also assess uniqueness using GPT-3.5.Source: A LLM PREAMBLES · p. 14, Figure 8 · p. 14

B DSG-1K DATASET DETAILS

• 3,180 model-generated images cover 1,060 prompts using Imagen* (Saharia et al., 2022), MUSE* (Chang et al., 2023), and SD v2.1.For uniform evaluation and human annotation, the authors also reuse images published by TIFA for TIFA160.Source: B DSG-1K DATASET DETAILS · p. 14
• Three human experts independently construct semantic tuples and dependencies for TIFA160, while the LLM pipeline generates tuples, questions, and dependencies throughout DSG-1k.The expert annotations are generated manually.Source: B DSG-1K DATASET DETAILS · p. 15
• Human annotators assess generated questions’ validity against images and answer valid questions Yes or No; automatic VQA models answer the same questions.Automatic answerers are mPLUG-large, Instruct-BLIP (Dai et al., 2023), and PaLI (Chen et al., 2023).Source: B DSG-1K DATASET DETAILS · p. 15
• 1,120 text-image pairs from seven models on TIFA160 receive human 1-to-5 Likert-scale alignment scores, used to measure correlation with QG/A judgments.The total comprises seven models evaluated on 160 prompts each.Source: B DSG-1K DATASET DETAILS · p. 15
• Question counts across broad and detailed semantic categories support the statistical adequacy of sample sizes for DSG-1k’s category-level analyses.Tables 7 and 8 present broad-category and detailed-category question counts, respectively; their supplied captions contain no numerical counts.Source: B DSG-1K DATASET DETAILS · p. 15, Table 7 · p. 15, Table 8 · p. 15

C INVALID VQA QUERY EXAMPLES

• VQA models can deny an entity’s existence yet affirm a dependent claim, with such inconsistencies more common when text-to-image models omit entities.One example denies a car exists but affirms that it is playing soccer. These invalid queries should not be posed; missing entities occur more frequently in weaker evaluated models such as minDALL-E and VQ-Diffusion.Source: C INVALID VQA QUERY EXAMPLES · p. 15, Figure 10 · p. 16

D.1 WAYS TO UTILIZE DEPENDENCY

• Standard DSG counts every child answer as NO whenever its parent question is answered NO.This is the dependency-handling method presented in the main text.Source: §D.1 · p. 16
• DSG (drop) excludes children of NO-answered parents from accuracy, whereas DSG (w/o dep) ignores dependencies and counts all answers equally.These alternatives distinguish dropping dependent questions from disregarding dependency annotations entirely.Source: §D.1 · p. 16
• The authors interpret the per-question findings as evidence that human raters implicitly consider dependencies, supporting more reliable and interpretable fine-grained evaluation.Aggregate correlation trends therefore do not tell the whole story.Source: §D.1 · p. 17
• Different dependency-handling strategies have no significant effect on correlations between VQA accuracy and human 1-5 Likert judgments.The correlation measures are Spearman’s ρ / Kendall’s τ.Source: Table 9 · p. 17, §D.1 · p. 17
• Using dependencies with DSG or DSG (drop) substantially improves per-question VQA–human matching accuracy.This contrasts with the lack of significant effects on aggregate correlations.Source: Table 10 · p. 17, §D.1 · p. 17

D.2 BY-CATEGORY ABLATION FOR TIFA, VQ2A, AND DSG

• The TIFA160 by-split ablation compares TIFA, VQ2A, and DSG using PaLI and full human annotations for VQA–Likert correlations.It measures Spearman’s ρ / Kendall’s τ against human 1-5 Likert scores. Splits cover general-purpose COCO, spatial-relation PaintSkill, attributional failures in PartiPrompts (per Yu et al. (2022)), and compositionality in DrawBench.Source: Table 11 · p. 17, §D.2 · p. 17

Single-summary metric context

• 5 single-summary metrics—BLEU-4, ROUGE-L, METEOR, SPICE, and CLIPScore—provide comparison context, with some outperforming TIFA or VQ2A in particular configurations.Caption-based metrics use BLIP-2 Captioning, while CLIPScore uses CLIP ViT-B/32. Results from Hu et al. (2023) are not directly applicable because their human ratings came from different raters.Source: E ADDITIONAL QG/QA/T2I EVALUATION… · pp. 17–18

Single-summary metric context / T2I evaluation

• DSG achieves much higher correlations with human ratings than the evaluated single-summary metrics across all VQA modules.Additional T2I analyses report average Likert (1–5) scores on TIFA160 and fine-grained VQA–human correlations and accuracies. Category correlations require ≥30 samples and p < 10^-4.Source: Table 12 · p. 18, E ADDITIONAL QG/QA/T2I EVALUATION… · p. 18

QA evaluation / QG evaluation

• VQA–human answer-match accuracy broken down by DSG-1k data source follows the trends observed in category-level evaluations.The analysis covers three T2I models using DSG-PaLM2 questions. QG evaluation also presents examples where human and GPT-3.5 uniqueness judgments disagree.Source: E ADDITIONAL QG/QA/T2I EVALUATION… · p. 18, Table 13 · p. 18

F EXPERIMENTS ON MIXED QA EVALUATION MODELS

• Counting experiments compare direct VQA with OWLv2-base-patch16 across TIFA, VQ2A, and PaLI on CountBench (Paiss et al., 2023).The object-detection alternative answers yes when the number of detected bounding boxes for the target object equals the requested count.Source: F EXPERIMENTS ON MIXED QA EVALUATION… · pp. 18–19
• Object detection slightly lowers VQA–human alignment for counting, whereas OCR noticeably improves text-rendering alignment, though the improvement is not categorically substantial.The comparisons measure VQA–human matching accuracy on CountBench and DrawText, respectively.Source: F EXPERIMENTS ON MIXED QA EVALUATION… · p. 18, Table 17 · p. 19, Table 18 · p. 19
• Text-rendering experiments compare direct VQA with EasyOCR (Jaided AI, 2023), integrated per VPEval (Cho et al., 2023b), on DrawText (Liu et al., 2023).The OCR alternative answers yes when the target text is captured.Source: F EXPERIMENTS ON MIXED QA EVALUATION… · pp. 18–19

G PSEUDOCODE OF T2I EVALUATION WITH DSG.

• The DSG pseudocode uses an LLM to generate atomic semantic tuples, their parent dependencies, and corresponding natural-language questions from the prompt.The generated structures are dictionaries indexed by tuple identifiers.Source: G PSEUDOCODE OF T2I EVALUATION WITH DSG · p. 20
• Image evaluation scores each VQA answer as one for yes and zero otherwise, then averages the dependency-filtered question scores.VQA receives the generated image and each DSG question.Source: G PSEUDOCODE OF T2I EVALUATION WITH DSG · p. 20
• DSG dependency filtering zeroes a question’s score if any parent question receives a negative answer.This filtering occurs after VQA answering and before final averaging.Source: G PSEUDOCODE OF T2I EVALUATION WITH DSG · p. 20

• Human answers and ratings are collected through interfaces exemplified in Figures 11 and 12.Source: H HUMAN EVALUATION SETUP · p. 20
• Figure 11 shows the interface for per-question human judgment annotation.Source: Figure 11 · p. 21
• 1-5 Likert consistency ratings are collected through the per-item human judgment interface shown in Figure 12.Source: H HUMAN EVALUATION SETUP · p. 20, Figure 12 · p. 21