PaperlayerView exact revision on arXiv

Source-linked AI summary

DEPICT: Scoring Text-to-Image Alignment by Answer Agreement

Vasco Ramos, Sandra Godinho Silva, Joao Magalhaes, Ricardo Rei, Pedro Henrique Martins

arXiv:2610.03617v1cs.CV

TL;DR

Fine-grained image-text alignment is important as text-to-image models improve, but holistic and decomposed metrics each have limitations. DEPICT uses caption-weighted agreement between image- and caption-based answers and merges it with holistic scoring. Across five benchmarks and eleven backbones, it matches or exceeds fine-tuned evaluators while avoiding fixed-yes failures.

  • Problem

    As text-to-image models improve, alignment metrics must detect fine-grained errors, but holistic and decomposed approaches have complementary strengths and neither dominates across benchmarks.

    Source: §1 · p. 1

  • Method

    DEPICT scores caption-generated questions by agreement between image-based and caption-only answers, weights them by caption commitment, and merges the decomposed score with a holistic score.

    Source: §3 · p. 3

  • Results

    Across five benchmarks and eleven backbones, expected agreement and term merging match or exceed fine-tuned evaluators, hold across backbones, and avoid fixed-yes failures.

    Source: §5 · p. 5

  • Takeaways & Limitations

    The merged score correlates better with human ratings than either holistic or decomposed scoring alone on every human-judgment benchmark tested.

    Source: §1 · p. 2

    Decomposition brings no gain on short prompts hinging on compositional word order, and a shared backbone can make shared-prior hallucinations appear to agree.

    Source: §6 · p. 9

Abstract

from arXiv · show

Image-text alignment is a core problem in computer vision with applications in caption evaluation, hallucination detection, data curation, and the benchmarking of text-to-image (T2I) generators. As T2I models improve, benchmarking has become demanding, requiring metrics capable of finding a series of issues like missing objects, swapped attributes, miscounts, and ignored negations. Recent work addresses this by fine-tuning evaluators on preference data or by prompting a vision-language model, either holistically with the caption or with decomposed verification questions. However, existing approaches fall short: fine-tuned metrics remain bound to one backbone and training distribution; holistic metrics miss fine-grained details; and decomposed metrics rely on a fixed-YES assumption that penalizes faithful images whenever that assumption fails. In contrast, we propose DEPICT, a training-free metric that replaces fixed reference answers with expected agreement between image-based and caption-only answers, weighting questions by how decisively the caption determines them. By replacing fixed references, our agreement rule increases negation accuracy from 19% to 88%. To recover the context lost during decomposition, DEPICT merges this agreement score with a holistic score. We evaluate DEPICT on five benchmarks and eleven backbones from three model families and find that it surpasses all training-free metrics and exceeds fine-tuned evaluators on two out of three human-correlation benchmarks.

1 Introduction

As text-to-image generators improve, alignment metrics must detect fine-grained errors, yet holistic and decomposed scoring each have limitations. DEPICT combines answer agreement with holistic scoring, and its evaluation spans multiple backbones and benchmarks.

1 Introduction

  • Fixed-yes decomposed metrics fail when the expected answer is wrong, while holistic scoring preserves global context but can miss fine-grained details.The paper describes alignment errors including dropped objects, swapped attributes, miscounts, and ignored negations.Source: §1 · pp. 1–2
  • DEPICT scores caption-derived assertions by agreement between image-based and caption-only answers, then merges this decomposed score with holistic scoring.The authors report that the merged score correlates better with human ratings than either approach alone across every human-judgment benchmark they test.Source: §1 · p. 2
  • The paper presents a comparison of training-free alignment metrics across backbones from several model families and multiple benchmarks, with paired confidence intervals for every difference.The authors frame this as, to their knowledge, the first such study.Source: §1 · p. 2
  • DEPICT consistently matches or beats training-free baselines across backbones and benchmarks, and exceeds the strongest fine-tuned evaluator on its own backbone on most human-judgment benchmarks.The study compares training-free metrics across backbones from several model families and multiple benchmarks, with paired confidence intervals for every difference.Source: §1 · p. 2

2 Related Work

Prior alignment metrics use global similarity, fine-tuning, or training-free VLM prompting, with holistic and decomposed approaches offering different strengths. DEPICT replaces static decomposed references with answer agreement and merges the decomposed and holistic approaches.

2 Related Work

  • Embedding-based metrics such as CLIPScore and BLIP2Score rely on global image-text similarity and struggle with compositionality, correlating weakly with human judgments.The passage attributes this behavior to coarse caption-matching pretraining and describes these metrics as bag-of-words matchers.Source: §2 · p. 2
  • Fine-tuned evaluators learn from preference data or human ratings, but their learned scores remain bound to a backbone and training data.DynEval is described as the strongest such evaluator and as the matched-backbone reference for this study.Source: §2 · p. 2
  • Training-free metrics query off-the-shelf VLMs; holistic scoring reads a single yes probability for the whole caption, while decomposed methods score per-question answers against fixed references.VQAScore captures overall context but remains blind to fine-grained details; Gecko and Soft-TIFA use per-question probabilities while retaining fixed references.Source: §2 · pp. 2–3
  • DEPICT replaces fixed references with agreement between text-only and visual answers and is the first method described here to merge decomposed and holistic scoring.The overview depicts agreement-based decomposed scoring combined with a holistic score.Source: §2 · p. 3, Figure 2 · p. 3

3 Measuring Alignment Between Text and Vision

DEPICT scores caption-image alignment using question-level answer agreement and caption-based commitment weights, then combines that decomposed score with a holistic score. Its questions are generated directly from the caption, while soft answer probabilities support continuous scoring.

Generating questions from the caption

  • Questions are generated directly from the full caption, each targeting one verifiable detail, without intermediate tuple parsing.The question generator sees the whole caption, avoiding possible parsing errors and retaining context that tuple extraction discards.Source: §3 · p. 3

Continuous soft scoring

  • Soft decomposed scoring uses image-answer probabilities without renormalizing over yes/no, so probability mass elsewhere lowers the score.This penalizes off-target outputs without forcing a binary answer.Source: §3 · p. 3

From a fixed reference to expected agreement

  • Agree scores expected image-text answer agreement across yes and no outcomes, unlike Faith’s image-side yes probability and Confirm’s positive-agreement-only rule.Agree is symmetric across modalities and reduces to Faith or Confirm under the stated special cases.Source: §3 · pp. 3–4

Commitment Weighting

  • Caption commitment weights each question by how decisively the caption answers it, so uncertain or off-target questions contribute less or vanish.This acts as a soft quality filter and removes the need for post-hoc pruning.Source: §3 · p. 4

Decomposed Score Aggregation

  • The decomposed score is the commitment-weighted mean of per-question scores, with epsilon preventing division by zero.Source: §3 · p. 4

Holistic Answering and Merge

  • DEPICT combines a commitment-weighted decomposed score with a holistic whole-caption score to retain both fine-grained details and macro-level context.The holistic score is the VLM’s probability of answering yes to the entire caption; λ balances it with the decomposed score.Source: §3 · p. 4

4 Experimental Setup

The experiments measure human-score correlation and diagnostic accuracy across multiple training-free metrics, fine-tuned evaluators, datasets, and VLM backbones. Training-free methods are compared across eleven backbones, while fine-tuned baselines remain tied to their training backbones.

Datasets, metrics, and statistics

  • Human correlation is measured on GenAI-Bench, TIFA160, and RichHF-18K using SRCC and PLCC; diagnostics use Winoground accuracy and NegBench COCO-MCQ.Score differences are paired, with BCa 95% intervals from 10,000 item-resampled bootstrap replicates.Source: §4 · p. 4

Baselines

  • The baseline set includes training-free embedding metrics, VLM-based holistic and decomposed metrics, and fine-tuned evaluators tied to their training backbones.Representative VLM baselines are DSG (Cho et al., 2024), Soft-TIFA, and VQAScore (Lin et al., 2024).Source: §4 · p. 4

Backbones

  • Training-free methods run on eleven backbones spanning Qwen3.5, InternVL3.5, and Gemma-4 families, with scale, generation, and architecture variants.Qwen3-VL-4B provides a matched comparison against DynEval.Source: §4 · p. 4

Implementation Details

  • Question generation and scoring use the same backbone; question counts vary by caption, and all training-free methods receive identical images, prompts, and decoding settings.Commitment weighting, no pruning, arithmetic aggregation, epsilon = 10^-3, and λ = 0.5 are fixed a priori rather than tuned on benchmarks.Source: §4 · p. 5

5 Results and Discussion

Across five benchmarks and eleven backbones, DEPICT matches or exceeds fine-tuned evaluators and holds up across backbones. The results attribute gains to answer agreement, merging, and scoring choices, with different effects across benchmarks and components.

5 Results and Discussion

  • Across five benchmarks and eleven backbones, expected agreement and term merging match or exceed fine-tuned evaluators, hold across backbones, and avoid fixed-yes failures.DEPICT is best or second best across all datasets and metrics in Table 1.Source: §5 · p. 5, §5.1 · p. 5

5.1 Does Depict compete with fine-tuned evaluators?

  • On the matched Qwen3-VL-4B backbone, DEPICT surpasses the strongest fine-tuned baseline on GenAI-Bench and RichHF, trailing only on TIFA160.Swapping VQAScore to Qwen3-VL-4B also raises its SRCC by 0.075 on GenAI-Bench and 0.124 on RichHF; moving DEPICT to Qwen3.5-27B adds 0.06–0.10 in every column.Source: §5.1 · pp. 5–6

5.2 Where do the gains come from?

  • Agree narrows the GenAI-Bench SRCC gap to VQAScore from 0.15 to 0.03 and raises NegBench accuracy to 88.3%; merging improves correlation further.The fixed-backbone comparison holds images, prompts, and decoding constant; Agree differs from Faith only in scoring, and DEPICT adds the holistic term.Source: §5.2 · p. 6, Table 2 · p. 6
  • DEPICT matches or improves on whichever term suits each correlation benchmark, with significant gain on GenAI-Bench; on Winoground it stays within VQAScore’s confidence interval.The merged metric’s lowest-scored items are rated worse by humans than VQAScore’s on every benchmark and backbone, while their top tails coincide.Source: §5.2 · p. 6, Table 2 · p. 6

5.3 Why do agreement and merging work?

  • Fixed-yes metrics fall below chance on negated items, while DEPICT is accurate across all three templates; merging reaches 60% accuracy on contested pairs versus 53% for holistic and 47% for decomposed scoring.Negated-caption questions expect “no” 87% of the time, compared with 4% elsewhere; the two terms disagree on 19% of GenAI-Bench item pairs.Source: Figure 3 · p. 6, §5.3 · p. 7

5.4 How do metrics behave across backbones?

  • Across eleven backbones, DEPICT matches or beats the strongest baseline in 54 of 55 backbone–benchmark pairs, winning 43 times, including 23 statistically significant wins.On the Qwen3.5 size ladder, NegBench accuracy stays at 85–87% across sizes, while all methods gain similarly on Winoground.Source: Figure 4 · p. 7, §5.4 · p. 8

5.5 Which components matter?

  • Tuple extraction harms all benchmarks and yields 4.1% negation-subset accuracy; pruning lowers performance across benchmarks, while Agree raises NegBench accuracy 4.6× over Faith.On Winoground, tuple extraction costs 21 group-accuracy points; discretizing Faith costs 16.3 points and 0.033 SRCC, while Agree moves Winoground by at most one point.Source: §5.5 · p. 8

6 Conclusions

DEPICT combines two-sided answer agreement with a holistic score, addressing trade-offs among fine-tuned, holistic, and decomposed evaluators. It matches or exceeds evaluators across five benchmarks and eleven backbones, though shared backbone priors and short compositional prompts remain limitations.

6 Conclusions

  • DEPICT combines agreement between image-based and caption-only answers with a holistic score, addressing trade-offs among fine-tuned, holistic, and decomposed metrics.Agreement raises negation accuracy from 19% to 88%.Source: §6 · p. 8
  • On a matched Qwen3-VL-4B backbone, DEPICT beats DynEval-4B on GenAI-Bench (0.630 vs. 0.595 SRCC) and RichHF (0.607 vs. 0.546), but trails on TIFA160 (0.802 vs. 0.691).Source: §6 · p. 8
  • Decomposition brings no gain on short prompts hinging on compositional word order, and shared backbone priors can make the same hallucination look like agreement.Enabling reasoning on answers improves results on the short-prompt case; using different backbones would decouple the priors but require hosting two models.Source: §6 · p. 9

A.1 Benchmarks

The benchmarks cover human-rated alignment and diagnostic caption–image comparisons, with separate prompts for question generation and image-based versus caption-only answers. DEPICT’s evaluation cost is higher in forward passes than Soft-TIFA’s, but the measured latency gap is smaller and cached multi-image evaluation reduces marginal cost.

A.1 Benchmarks

  • GenAI-Bench, TIFA160, and RichHF-18K use human ratings; SRCC is reported on raw scores and PLCC after a four-parameter logistic fit.GenAI-Bench and TIFA160 use 1–5 alignment ratings, while RichHF-18K uses its per-image misalignment score.Source: §A.1 · p. 13, Table 4 · p. 13
  • Winoground tests image–caption matching with reordered word combinations, while NegBench selects the correct caption from affirmative, negated, and hybrid candidates.NegBench pairs 5,914 COCO val2017 images with four captions each; chance accuracy is 25%.Source: §A.1 · p. 13, Table 4 · p. 13
  • Captions yield a mean of 2.71 generated questions; descriptive captions average about five, while NegBench’s templated captions average 1.92.The longest RichHF caption yields 32 questions, and the epsilon in Eq. 5 guards against captions yielding none.Source: §A.1 · p. 13

A.2 Prompts

  • The prompts separately generate caption-derived questions, image-based answers, and caption-only reference answers.The visual-answer prompt receives the image and one question; the text-only prompt receives the caption and one question.Source: Figure 5 · p. 13, Figure 6 · p. 14, Figure 7 · p. 15

A.3 Cost

  • DEPICT uses 2N + 2 forward passes per pair (about 12.9 on average) versus Soft-TIFA’s N + 1 (about 6.5), but measured latency is only 1.36× Soft-TIFA’s.On GenAI-Bench, DEPICT takes 0.492 seconds per pair and Soft-TIFA 0.362 seconds; VQAScore takes 0.099 seconds.Source: §A.3 · p. 16, Table 5 · p. 16, Table 6 · p. 16
  • Question generation dominates runtime, while cached question generation and text-only answers reduce DEPICT’s marginal cost per additional image to N + 1 visual passes.That marginal cost is virtually identical to Soft-TIFA’s N passes.Source: §A.3 · p. 16, Table 5 · p. 16

B.1 Full backbone sweep

The backbone sweep finds that benchmark performance varies with size, architecture, and model generation. Across the additional analyses, DEPICT remains robust on negation, gains vary by skill and question commitment, and the merged score performs well across a broad range of merge weights.

B.1 Full backbone sweep

  • Winoground improves with size, correlation gains are modest and not always monotonic, and DEPICT’s NegBench accuracy stays flat; at 27B, Qwen3.8 gives its largest margins.Gemma-4’s 12B model without a vision encoder is weakest on every benchmark and the only backbone where DEPICT significantly loses to a baseline, VQAScore on GenAI-Bench.Source: §B.1 · p. 16, Table 7 · p. 17, Table 8 · p. 18

B.2 Negation across backbones

  • On further backbones, fixed-yes metrics score at most 0.23 for DSG and 0.10 for Soft-TIFA on negation items, while DEPICT scores at least 0.78 on negation and 0.72 on hybrid items.Table 9 reports NegBench accuracy, with chance at 0.25.Source: §B.2 · p. 20, Table 9 · p. 20

B.3 GenAI-Bench by skill

  • On GenAI-Bench, DEPICT improves over VQAScore for every skill tag, with gains of +0.067 on differentiation and +0.066 on negation.The gain is three times larger on advanced-skill prompts (+0.040) than on basic-only prompts (+0.013).Source: §B.3 · p. 20, Table 10 · p. 20

B.4 Merge-weight sensitivity

  • Across all three correlation benchmarks, the merged score exceeds either component over λ ∈ [0.32, 0.82]; λ = 0.5 is within 0.003 SRCC of each benchmark’s best weight.Figure 9 compares SRCC on the correlation benchmarks and accuracy on NegBench; λ = 0 is decomposed-only and λ = 1 is holistic-only.Source: §B.4 · p. 20, Figure 9 · p. 21

C.1 Tuple extraction

  • Tuple extraction leaves 47% of NegBench negation captions without tuples, and tuple-derived Winoground questions overlap more than directly generated questions.Negation accuracy falls to 4.1%; Winoground question-set Jaccard overlap is 0.44 for tuples versus 0.27 for direct generation.Source: §C.1 · p. 21, Table 11 · p. 21

C.2 When commitment weighting acts

  • Commitment weighting improves DEPICT in 36 of 55 evaluations and yields a 0.68-point gain on items with the least committed questions, versus 0.02 on the most committed quartile.It changes a score only when question commitment varies across the item; weighting most often helps backbones whose text channel hedges more often.Source: §C.2 · p. 21

C.3 Merge ablation

The merge ablation shows that results depend on the decomposed scoring rule, while adding chain-of-thought reasoning has different effects across benchmarks. Reasoning can improve Winoground group accuracy, but the authors keep it off to retain graded probabilities and avoid extra token cost.

C.3 Merge ablation

  • On GenAI-Bench, the Soft-TIFA merge scores 0.696 versus 0.698 for VQAScore alone, while DEPICT gains 0.024; on NegBench, the Soft-TIFA merge scores 62.2 versus 84.7 for VQAScore.The two merges tie on TIFA160 and Winoground; the Soft-TIFA merge is slightly higher on RichHF (−0.008, not significant).Source: §C.3 · pp. 21–22, Table 13 · p. 22

C.4 Reasoning in the answering step

  • Reasoning variants score at or below reasoning-off on GenAI-Bench and NegBench, but add up to +5.5 Winoground group points.On Winoground, reasoning reduces images where the holistic term rejects both captions from 18.5% to 7.4%; the decomposed and holistic errors also become less correlated (ϕ falls from 0.52 to 0.43).Source: §C.4 · p. 22, Table 14 · p. 22

D.1 Answer distributions

Caption-only answers usually reflect the polarity of negated captions, and the distribution analysis shows that negation templates account for most expected-no answers. DEPICT’s scores are more evenly spread than the compared metrics, with its lowest-scored items rated worse by humans.

D.1 Answer distributions

  • On NegBench, questions whose caption-implied answer is no make up 34.7% overall; by template, the shares are 86.8% for negation, 47.1% for hybrid, and 0.4% for affirmative captions.Undecided questions are rare, at no more than 1.4%.Source: §D.1 · p. 23

D.2 Reference reliability

  • Caption-only answers achieve 97.6% overall polarity accuracy on NegBench negation captions; 3.2% of cases count as errors, some potentially due to parser misattributions.The parsed sample covers 80.1% of negation captions.Source: §D.2 · p. 23

D.3 Score distributions

  • On Qwen3.5-27B, 60–64% of DEPICT scores fall between 0.2 and 0.8, and its lowest-scored 5% are rated worse by humans than VQAScore’s on every benchmark and backbone.Figure 10 shows score-bin fractions for Qwen3.5-27B and Qwen3-VL-4B; the top tails coincide.Source: §D.3 · p. 23, Figure 10 · p. 23

E Qualitative examples

Qualitative selection examples show Depict succeeding where baselines fail on negation and attribute or spatial binding, while Fig. 18 exposes failures in both selection tasks.

E Qualitative examples

  • Depict selects human-preferred images or correct captions where baselines fail, most clearly on explicit negation and attribute or spatial binding.The examples cover choosing images for captions and captions for images.Source: E Qualitative examples · p. 24, Figure 14 · p. 25, Figure 15 · p. 25
  • Depict gives graded scores, whereas DSG often ties several candidate images at 1.0.Source: E Qualitative examples · p. 24
  • On NegBench #2034, Depict ranks the correct caption third of four (0.22 vs 0.53) after image and holistic channels misread the bottle negation.The image channel assigns the bottle assertion pv ≈0.7; holistic scores are 0.41 for the distractor and 0.15 for the correct caption.Source: E Qualitative examples · p. 24, Figure 18 · p. 26
  • On GenAI-Bench #247, all methods prefer Midjourney (human rating 4.0) over DALL-E 3 (5.0), and Depict’s holistic scores are 0.95 vs 0.69.The decomposed scores mainly differ on whether crystals light the scroll: agreement is 0.98 for Midjourney and 0.35 for DALL-E 3; neither image is judged to show exactly two crystals.Source: E Qualitative examples · p. 24, Figure 18 · p. 26

Related papers

Loading 2610.03617v1…