NegT2IBench: When Negation Changes the Picture. A Polarity Benchmark for Text-to-Image Models

Motivation

• Existing text-to-image benchmarks focus on whether requested content appears and largely overlook whether models respect what prompts forbid.Negation evaluation must also address accidental success, object omission, prompt complexity, and evaluator errors.Source: §1 · p. 1

Benchmark design

• NEGT2IBENCH uses plausible attributes and relations, fixed prompt templates, required object presence, and separate checks for each requirement to address negation-specific evaluation challenges.Prompts with the same statement count keep object counts the same while varying polarity; the benchmark covers two attribute and four relation categories.Source: §1 · pp. 1–2

Results

• Nine of eleven models score lower on a single negated statement than on a single positive one, with color dropping from 85.0% to 31.3% while proximity changes from 45.8% to 43.2%.At the highest tested complexity, no model exceeds 8.4%; 41.5% of failed statements render what the prompt forbids.Source: §1 · p. 2

Contributions

• The benchmark contains 4,800 prompts over 79 object classes, six categories, and eight polarity settings, scored using detection and geometry.It evaluates attributes and relations as well as negated objects, while requiring corresponding objects to remain present.Source: §1 · pp. 2–3

Text-to-image models and compositional evaluation

• Existing compositional benchmarks largely assess whether requested content appears; NEGT2IBENCH builds on GenEval [3] and PosEval [25] while reporting human agreement by category and polarity.Detector-based benchmarks can face template contamination [32] and wording sensitivity [33].Source: §2 · p. 3

Negation in text-to-image generation

• Prior negation benchmarks do not compare positive and negated attribute and relation prompts while requiring every named object to remain present.Negative-prompt steering excludes content but does not test whether a model follows negation written in the prompt.Source: §2 · p. 3

Judging negation

• Image-text scores do not establish negation reliability, and VQA judges can hallucinate absent details [7] [56] or lose up to 18.8 κ points when negations are phrased as negations [57].NEGT2IBENCH instead checks each element with dedicated perception components and reports human agreement by category and polarity.Source: §2 · p. 3

3 The NEGT2IBENCH Benchmark

• The eight settings vary positive and negated statement counts, denoted P:N, while all named objects remain required in either polarity.For relation prompts, M statements name M +1 objects; changing polarity leaves object counts fixed.Source: §3 · p. 3
• NEGT2IBENCH’s six categories each contain 800 prompts, with 100 prompts per polarity setting across eight settings and 79 object classes.Matched examples keep the same objects and value while changing only polarity.Source: Table 1 · p. 4, §3 · p. 4
• An LLM selects plausible combinations from fixed lists, rule-based templates standardize wording, and the benchmark balances underused object, attribute, and relation types.The construction adapts GenEval’s color vocabulary and position rules [3], OVAD material values [59], and PosEval relation encoding [25].Source: §3 · p. 4

3.1 Detector-based evaluation

• Each prompt requirement is checked separately, and verifying every named object first prevents satisfying a negated statement by omitting its object.RF-DETR-XL detects objects, SAM 2 refines masks, and Depth-Anything V2 estimates depth for front-behind relations.Source: §3.1 · p. 4, Figure 2 · p. 5
• SigLIP 2 scores masked crops for color and material, while relation rules test whether positive relations hold and negated relations do not.Ambiguous relation measurements fail positive statements but pass negated ones when the denied relation is not found.Source: §3.1 · p. 4
• Image accuracy requires every object, count, and statement to be correct; prompt accuracy counts prompts with at least one correct image among four attempts.Failed statements are assigned one of five causes for diagnosing model failures.Source: §3.1 · pp. 4–5

Human reference and evaluators

• The human reference comes from three labels per image across 600 images, with majority votes producing binary image labels and continuous element-correctness scores.The evaluator comparison includes VQA judges in whole-prompt and per-element modes, plus image-text similarity and reward scores.Source: §4 · p. 5

Negative phrasing effect

• Asking negated statements in negated form lowers all judges’ κ by 1.3 to 18.8 points, while NEGT2IBENCH beats every judge under whole-prompt and negated phrasing.Five judges also lose 4.5 to 6.9 points of τ under negated phrasing.Source: §4 · pp. 5–6

Agreement with humans

• NEGT2IBENCH reaches κ = 72.0 and τ = 65.1 against human labels, exceeding human inter-annotator κ = 65.6 and the best judge’s τ = 63.3.Only Qwen3-VL-32B scores higher on κ, at 74.3; NEGT2IBENCH has recall 85.0 versus the best judge’s 82.0 and precision 79.2 versus 84.1.Source: §4 · p. 5, Table 2 · p. 6

Cost

• NEGT2IBENCH uses 6.3 GiB of GPU memory and scores an image in 0.78 s; similarly agreeing judges use 16.8 to 73.0 GiB.The detector evaluator has 1.24B parameters and runs once per image, while judges ask one question per element.Source: §4 · p. 6

5 Benchmarking text-to-image models

• Nine of eleven configurations score higher on 1:0 than 0:1; the best score falls from 76.3% at Load 1 to 8.4% at Load 4.The results distinguish a negation penalty at matched load from difficulty satisfying more statements, regardless of polarity.Source: §5 · pp. 6–7, Table 3 · p. 7
• 53.7 points for color (85.0% asserted to 31.3% denied) was the largest Load 1 drop; proximity barely changed (45.8% vs. 43.2%), while depth reversed (41.7% vs. 44.0%).A larger drop does not by itself establish that a category is harder to negate, because it can partly reflect a higher positive baseline.Source: §5 · p. 7
• A model's positive-to-negated drop cannot rank negation ability, because a weaker positive baseline caps how much it can lose.Negated statements can also pass by chance when the model ignores negation; an omitted-attribute or relation control is left for future work.Source: §5 · p. 7
• Negated violations account for 41.5% of failed statements, and negated statements fail more often than positive ones within every mixed setting.Negated violations are the largest failure group; their share is 61% for color and 34–41% for the four relations.Source: §5 · p. 7
• 19.0% of passing negated relation statements pass only because no relation was resolved, mostly in depth (55.5%); this explains depth's Load 1 reversal.Such passes are valid under the benchmark rule but rest on weaker evidence than a pass where an alternative relation is established.Source: §5 · pp. 7–8
• NegationCLIP raises SD 1.5 overall accuracy from 14.5% to 14.9%, but differing guidance scales make this comparison suggestive rather than causal.Both SD 1.5 variants score higher at 0:1 than at 1:0 and remain far below the other models.Source: §5 · p. 8

6 Conclusion

• Nine of eleven configurations score lower on one negated than one positive statement; aggregate compositional scores hide this distinct ability.The detector-based evaluator agrees with human labels comparably to much larger vision-language judges while using a fraction of their GPU memory.Source: §6 · pp. 8–9

A.1 Prompt construction and representative examples

• At a fixed statement load, changing polarity preserves object count; relation prompts use one shared subject and distinct targets, while negation keeps all named objects required.For example, the two-statement settings pos2 neg0, pos1 neg1, and pos0 neg2 have the same statement count.Source: §A.1 · p. 14
• The taxonomy uses 79 detector-compatible COCO object classes, excludes person, and treats size as a comparative relation rather than an intrinsic attribute..Source: §A.1 · p. 14
• An LLM filters fixed-vocabulary combinations for plausible positive forms, after which deterministic templates keep wording stable across polarity conditions.Selection balances use of objects, values, and predicates rather than simply following selector preference.Source: §A.1 · p. 14

A.2 Dataset statistics and balancing

• The final corpus has 4,800 prompts across six categories and eight settings, with 100 prompts per category-setting cell and all 79 eligible COCO classes in each category.Object counts are close to the most-even allocation achievable, while predicate and attribute-value vocabularies are approximately uniformly distributed.Source: §A.2 · p. 14, Figure 5 · p. 16, Figure 6 · p. 17

B.1 Reference labels and human inter-annotator score

• Across 600 images labeled by 36 annotators, negated statements have lower agreement than affirmative ones: κ = 60.2 and 70.2% unanimous versus κ = 73.0 and 88.3%.The leave-one-out inter-annotator score is a conservative reference, not an upper bound on evaluator agreement with the three-annotator consensus.Source: §B.1 · p. 15, Table 5 · p. 15

B.2 Evaluators’ scores

• Per-element judges assess required objects and statements separately, with expected answers adjusted for polarity; NEGT2IBENCH reports the fraction of elements its detectors mark satisfied.This shared decomposition lets evaluators be compared on the same requirements and identifies which requirement failed.Source: §B.2 · p. 16

B.3 Measures

• Binary evaluation uses accuracy and Cohen’s κ against human labels, while continuous evaluation compares evaluator scores with human scores using rank-based measures.Similarity and reward scores are excluded from binary evaluation because no single threshold applies equally to them.Source: §B.3 · p. 16
• NEGT2IBENCH leads on Kendall τ (65.1 vs. 63.3 for Qwen3-VL-32B) but trails on Spearman ρ (72.8 vs. 76.5) and pairwise accuracy (91.7 vs. 94.7).The three metrics order evaluators differently because they treat ties and rank displacements differently; intervals use paired bootstrap resamples of whole images.Source: §B.3 · pp. 16–17
• PickScore, the best of six similarity and reward scores, reaches τ = 27.7, versus 38.8 for the weakest VQA judge other than mPLUG.NegationCLIP improves over CLIPScore only from 13.1 to 16.6; neither similarity nor preference measures whether a negated requirement holds.Source: §B.3 · p. 17

B.4 Cost protocol

• At 4.66 elements per image, NEGT2IBENCH takes 0.78 s per image and uses 2.7 to 11.6 times less GPU memory than the compared judges.It is faster than Qwen3-VL-32B and InternVL3.5-38B, but slower than the 8B judges with comparable agreement; measurements use one H200 NVL GPU without batching.Source: §B.4 · pp. 17–18

B.5 Error structure

• NEGT2IBENCH accepts 36.8% of images, with 46 false passes and 31 false failures; Qwen3-VL-32B accepts 33.5%, with 32 false passes and 37 false failures.Humans accept 34.3% of images; NEGT2IBENCH rarely rejects images humans consider correct, while per-element mPLUG accepts only 8.8% and makes 170 false failures.Source: §B.5 · p. 19, Table 9 · p. 21

B.6 Sensitivity to how a requirement is phrased

• Every judge loses agreement when negated questions are phrased in negated form; the six instruction-tuned judges lose 7.8 to 18.8 points, while NEGT2IBENCH is unaffected.Qwen3-VL-32B flips answers on 77.5% of elements, compared with 40.7% for Qwen2.5-VL-7B, whose complementarity error is 0.589.Source: §B.6 · p. 19, Table 10 · p. 22

B.7 Agreement by polarity setting

• All six similarity and reward scores fall from τ = 20.5–49.1 at 1:0 to τ = −22.2–20.7 at 0:1, while NEGT2IBENCH stays between 48.1 and 88.6 across settings.Each setting has 70 to 90 images, leaving per-cell 95% intervals about 32 points wide, so the passage reports patterns shared by many evaluators.Source: §B.7 · p. 19, Table 7 · p. 20

B.8 Agreement by category

• Whole-prompt judges agree well on color (65.4 to 78.4) but less on relations (25.3 to 52.8); NEGT2IBENCH is more even across categories (54.4 to 78.5).Per-element questions recover part of the judges’ relation gap; the strongest judges mainly lead on color and depth, where dead zones most affect NEGT2IBENCH.Source: §B.8 · pp. 20–21

B.9 Per-element agreement

• On negated statements, NEGT2IBENCH reaches κ = 56.4, within the best judge’s interval at 60.7 and close to the 59.7 inter-annotator score.Its positive-to-negated agreement loss is 7.9 points, compared with 14.3 to 18.5 for the three strongest judges; every judge falls further when questions are phrased in negated form.Source: §B.9 · p. 21, Table 10 · p. 22

C.1 Evaluation rules

• Positive attributes pass when the requested value is in the SigLIP 2 confusion set; negated attributes pass when the denied value is absent.Color and material are classified from detector-masked crops; near-tied denied values are not treated as confidently absent.Source: §C.1 · p. 22
• Directional rules use refined-mask centroids and extents, while proximity uses minimum edge-to-edge mask distance and complementary near/far predicates.Directional predicates require a normalized component to exceed αdir = 0.45; proximity has no geometric dead zone.Source: §C.1 · pp. 25–26
• Comparative relations measure visible in-image size, while depth predicates use estimated disparity and require stronger evidence as objects are farther apart.Pairs without sufficiently decisive measurements may produce no comparative or depth predicate.Source: §C.1 · pp. 26–27
• A positive relation passes when its predicate fires; a negated relation passes when it does not, so an empty predicate set fails positives but passes negations.Directional, comparative, and depth rules have deliberate dead zones; proximity predicates are exact complements.Source: §C.1 · p. 27

C.2 Evaluator configuration

• RF-DETR-XL supplies detections, and SAM 2 refines masks only for relation measurements; the evaluator uses its current configuration for reported benchmark results.When no detection reaches the counting threshold, the most confident detection is retained for judging; excess counted instances are flagged.Source: §C.1 · p. 22, §C.2 · p. 27
• The evaluator ablation varies the detector, SAM refinement, and relation-rule constants while leaving the SigLIP 2 attribute configuration unchanged.The compared configurations are summarized in Table 15 and Table 16.Source: §C.2 · p. 27, Table 15 · p. 28, Table 16 · p. 28

C.3 Metrics and failure definitions

• Prompt accuracy measures complete-prompt correctness across generated images, and failure diagnostics classify statement- and prompt-level errors without defining another benchmark score.Failure categories include undetected objects, over-counts, positive violations, negated violations, and dead-zone failures.Source: §C.3 · p. 27

D Evaluator ablations

• C6 is used throughout NEGT2IBENCH, retaining RF-DETR-XL, SAM refinement, and current relation constants.The ablation compares eight configurations formed by varying detector, SAM refinement, and relation-rule constants.Source: D Evaluator ablations · pp. 28–29
• The geometric cut points and attribute confusion buffer were fitted on 140 human-annotated images, then fixed across configurations.All configurations use RF-DETR-XL, SAM 2-refined relation masks, SigLIP 2 for attributes, and Depth-Anything V2 for depth.Source: D Evaluator ablations · p. 28
• The ablation uses 140 of 600 annotated images with clear human-majority verdicts, so its point estimates are a compact diagnostic, not a precise configuration ranking.SAM refinement is retained in C6 to tighten object masks for geometric relation measurements.Source: D Evaluator ablations · pp. 28–29

E.1 Experimental setup

• Eleven model configurations generate four samples for each of the 4,800 prompts, totaling 211,200 images.Each configuration contributes 19,200 images.Source: §E.1 · p. 29
• Models use their native generation configurations rather than a forced equal inference-compute budget, so absolute comparisons also reflect inference schedule and output resolution.Within-model comparisons across benchmark settings do not share that stated caveat.Source: §E.1 · p. 29
• No model-specific prompt engineering is applied, and the CFG negative-prompt field is unused.The SD 1.5 and SD 1.5+NegationCLIP comparison also uses different guidance scales, so it is not a perfectly controlled encoder intervention.Source: §E.1 · p. 29

E.2 Additional results

• 475,200 statement evaluations span the benchmark, with 18 statements per prompt across eight settings, six categories, four samples, and eleven configurations.Of these, 280,067 are failed statements; the reported statement-level percentages use failed statements as their denominator.Source: §E.2 · p. 30
• 40.1% of incorrect generations contain only negated-statement violations, 25.9% only positive violations, and 13.9% violations of both polarities.Over-counting accounts for 10.5% and undetected objects for 9.6% of incorrect generations.Source: §E.2 · p. 30
• 11,907 of 62,790 successful evaluable negated relation statements are dead-zone passes (19.0%), equal to 7.5% of all negated relation statements.Proximity has no designed dead zone, whereas 55.5% of successful negated depth statements occur when no depth predicate is resolved.Source: §E.2 · p. 30
• Replacing SD 1.5’s text encoder with NegationCLIP changes overall image accuracy from 14.5% to 14.9%, but differing guidance scales make the comparison suggestive rather than causal.Both configurations score higher on pos0 neg1 than on pos1 neg0, yet remain well below other configurations overall.Source: §E.2 · p. 30

E.3 Additional qualitative examples.

• Figure 8 shows six prompts, one per semantic category, across eleven models, with evaluator pass/fail marks for complete-prompt satisfaction.The examples complement quantitative results by making requested and forbidden properties, and preservation of named objects, concrete.Source: §E.3 · p. 31

• Coverage is restricted to automatically verifiable color, material, and four spatial and size relations over 79 COCO classes.Source: F Limitations · p. 31
• Contact, interactions, and negated whole-object presence are not covered, while comparative relations use apparent rather than real-world size.Source: F Limitations · p. 31
• Prompts use one English negation form and link one subject to every target, leaving paraphrase, other-language, and richer-relation robustness untested.Source: F Limitations · p. 31
• The benchmark measures absence rather than understanding, so ignored negation can still pass some statements by chance.Source: F Limitations · p. 31
• Scores depend on off-the-shelf perception-model quality, and open-weight models were evaluated under native rather than matched-compute settings.Source: F Limitations · p. 31