PaperlayerView exact revision on arXiv

Source-linked AI summary

Seeing Isn't Believing: Uncovering Blind Spots in Evaluator Vision-Language Models

Mohammed Safi Ur Rahman Khan, Sanjay Suryanarayanan, Tushar Anand, Mitesh M. Khapra

arXiv:2604.21523v1cs.CVcs.CL

TL;DR

Evaluator VLMs are increasingly used to judge model outputs and shape training rewards, making their reliability important to assess. This paper introduces FOCUS, a perturbation-based benchmark across I2T and T2I, and finds substantial blind spots: pairwise comparison is most reliable, while single-answer scoring is weakest.

  • Problem

    Evaluator VLMs are increasingly used to judge model outputs and shape training rewards, making rigorous assessment of their reliability important.

    Source: §1 · p. 1

  • Method

    FOCUS uses targeted perturbations across I2T and T2I to assess four prominent Evaluator VLMs under single-answer, pairwise, and reference-guided paradigms.

    Source: §1 · p. 2, §2 · p. 3, §6 · p. 10

  • Results

    Across I2T and T2I, pairwise comparison is most reliable and single-answer scoring weakest, with single-answer T2I failure rates exceeding 50% in some cases.

    Source: §4.1 · p. 6, §6 · p. 10

  • Takeaways & Limitations

    Evaluator VLMs should not be relied on as standalone judges; evaluations should be complemented with human review or task-specific metrics.

    Source: §1 · p. 3, §4.8 · p. 10

    The evaluation measures directional score changes and assumes gold answers and images are reasonably accurate and relevant, rather than requiring perfect references.

    Source: §2.1 · p. 3

Abstract

from arXiv · show

Large Vision-Language Models (VLMs) are increasingly used to evaluate outputs of other models, for image-to-text (I2T) tasks such as visual question answering, and text-to-image (T2I) generation tasks. Despite this growing reliance, the reliability of these Evaluator VLMs remains under explored. In this work, we systematically evaluate the reliability of Evaluator VLMs across both I2T and T2I tasks. We introduce targeted perturbations that degrade output quality along key error dimensions, including object hallucinations, spatial reasoning, factual grounding, and visual fidelity. These perturbations test whether Evaluator VLMs can reliably account for these quality degrading errors in their evaluations. Using a comprehensive benchmark of over 4000 perturbed instances spanning 40 perturbation dimensions, we evaluate 4 prominent VLMs using single-answer scoring, pairwise comparison, and reference-guided paradigms. Our findings reveal that current VLM evaluators exhibit substantial blind spots: they often fail to detect perturbed outputs - in some cases exceeding 50%, struggle particularly with fine-grained compositional and spatial errors, and are often insensitive to hallucinated content that contradicts the input image. Pairwise comparison proves more reliable, though failure rates persist. These results highlight the unreliable nature of current Evaluator VLMs and urge caution in their deployment for benchmarking and development decisions. Code and data have been made publicly available.

1 Introduction

The introduction presents FOCUS, a benchmark for uncovering Evaluator VLM blind spots across image-to-text and text-to-image tasks, and reports that current evaluators often miss quality-degrading errors.

Motivation and implications

  • Evaluator VLMs are used for scalable evaluation and as training reward models, so unreliable judgments can misrank outputs or reinforce undesirable behavior.The reported blind spots caution against relying on these models as standalone judges.Source: §1 · pp. 1–3

Evaluation challenge

  • Reliable evaluation requires checking image grounding in I2T and prompt fidelity in T2I, including fine-grained objects, attributes, facts, plausibility, and rendered text.Prior work has examined correlation with human judgments, but the introduction argues that evaluators also require rigorous scrutiny for systematic blind spots.Source: §1 · pp. 1–2

Benchmark

  • FOCUS tests whether evaluators detect degradation caused by targeted perturbations across I2T and T2I outputs.The benchmark contains over 4000 expert-validated instances, built from 600 I2T and 750 T2I prompts with gold and perturbed responses.Source: §1 · p. 2

Results

  • Evaluator VLMs miss quality-degrading perturbations in some cases at rates exceeding 50%.Performance is notably worse for T2I than I2T, and pairwise comparison is the most reliable paradigm. Failures concentrate in fine-grained visual grounding, compositional reasoning, and physical plausibility; increased reasoning budgets do not consistently help, and errors identified in justifications may not affect final scores.Source: §1 · p. 2

Evaluation

  • The study evaluates four prominent VLMs using single-answer scoring, pairwise comparison, and reference-guided evaluation.It also explores multiple prompting strategies proposed in prior work and used in practice.Source: §1 · p. 2

2 FOCUS benchmark

FOCUS is a human-overseen benchmark for testing whether Evaluator VLMs detect controlled errors in outputs from image-to-text and text-to-image tasks. It samples prompts from existing benchmarks, defines task-specific perturbation categories, and combines automatic perturbation generation with human review.

2 FOCUS benchmark

  • FOCUS evaluates VLMs judging other models’ outputs across I2T and T2I, pairing inputs and gold outputs with perturbed answers or images to test whether evaluators account for controlled errors.Instances receive human oversight during prompt selection, perturbation-dimension definition, and perturbation creation.Source: §2 · p. 3

2.1 Prompt selection

  • I2T uses 600 image-prompt instances and T2I 750 prompts sampled from seven benchmarks each; gold outputs from GEMINI-3.1-PRO and GEMINI-3-PRO-IMAGE were manually reviewed.Gold outputs need only be reasonably accurate and relevant because the study tests directional score changes between perturbed and original outputs.Source: §2.1 · p. 3

2.2 Perturbation categories

  • I2T perturbations cover visual grounding, semantic interpretation, visual reasoning, and long-form generation errors.These include altered entities or spatial relations, contextual or cultural inconsistencies, logical or numerical errors, and text inconsistent with the image.Source: §2.2 · pp. 3–4
  • T2I perturbations cover visual fidelity, scene coherence, physical plausibility, and text-rendering errors.Examples include changed image attributes or layouts, stylistic mismatches, physical violations, and corrupted text in an image.Source: §2.2 · p. 4

2.3 Perturbation generation

  • Perturbations are generated automatically with task-specific instructions and then human-reviewed as valid, invalid, or score-invariant.Valid changes introduce subtle, meaningful errors warranting lower scores; invalid changes are obvious or nonsensical, while score-invariant changes should not be penalized.Source: §2.3 · pp. 4–5

3 Experimental Setup

The setup benchmarks evaluator VLMs on FOCUS using three established evaluation paradigms and progressively structured prompts, then measures perturbation detection across I2T and T2I tasks.

3 Experimental Setup

  • Four prompting strategies progressively add structure—Vanilla, Rubric/Rules, Axes, and Axes+Rubric/Rules—within setups based on common literature rather than new evaluation strategies.The strategies move from input and output alone to predefined axes with per-axis rubrics or rules.Source: §3 · p. 5
  • Single-answer scoring judges one response independently: I2T uses the prompt, image, and generated answer, while T2I uses the prompt and generated image.The paper describes this as the most commonly used paradigm in practice (Chen et al., 2024b; Pu et al., 2025).Source: §3 · p. 5
  • Pairwise comparison selects the better of two candidates, using the prompt and image plus two answers for I2T, or the prompt and two images for T2I.The paradigm is widely used in preference-based evaluation and reward modeling (Lin et al., 2025; Chen et al., 2024a).Source: §3 · p. 5
  • Reference-guided scoring compares a model output with a gold answer or image, providing an explicit quality anchor that may be unavailable for many open-ended tasks.I2T uses a gold answer; T2I uses a gold image.Source: §3 · p. 5

3.1 Metrics

  • For single-answer scoring, the metric is the share of instances whose score remains unchanged after perturbation; multi-axis evaluators count only relevant axes, and all must remain unchanged for a miss.Ideally, evaluators assign lower scores to perturbed outputs.Source: §3.1 · p. 6
  • For pairwise comparison, the metric is the share of instances where the evaluator fails to select only the gold output; reference-guided scoring counts perfect scores assigned to perturbed outputs.Pairwise evaluations swap gold and perturbed-output order and run twice to mitigate position bias (Zheng et al., 2023b); axis-based comparisons consider only relevant axes.Source: §3.1 · p. 6

4 Results and discussion

Across I2T and T2I, structured pairwise evaluation is generally most reliable for detecting perturbations, but evaluator performance depends on the model, task, prompting strategy, and reasoning budget. Evaluators remain less reliable for fine-grained errors and can misjudge valid alternatives or fail to reflect recognized errors in their scores.

4.1 Which evaluator paradigms and strategies are most reliable?

  • Pairwise comparison is most reliable for valid perturbations, single-answer scoring is weakest, and reference-guided evaluation lies between them across I2T and T2I.Axes and Axes+Rules perform best among the reported strategies; single-answer failure rates exceed 50% in some T2I cases.Source: §4.1 · pp. 6–7, Table 3 · p. 7

4.6 How robust are evaluators on score-invariant perturbations?

  • For score-invariant perturbations, single-answer scoring is most robust and pairwise comparison least stable, as forced choices can favor one of two equally good responses.Simpler prompts generally perform better; reference-guided scoring is more stable than pairwise comparison but can penalize valid alternatives that differ from the reference.Source: Table 3 · p. 7, §4.6 · p. 9

4.2 How does evaluator performance vary across VLMs?

  • GEMINI-3.1-PRO has the lowest failure rates in pairwise evaluation across both tasks, while model rankings vary more in I2T single-answer scoring.CLAUDE-OPUS-4.6 is competitive under structured I2T strategies, GPT-5.4 is generally slightly behind Gemini, and QWEN3.5-397B-A17B tends toward higher failure rates, especially in T2I.Source: §4.2 · p. 7

4.3 Which perturbation categories are hardest to detect?

  • Visual Grounding and Semantic Interpretation challenge I2T single scoring, Physical Plausibility challenges T2I scoring and comparison, and Text Rendering challenges T2I reference evaluation.Pairwise comparison reduces I2T failures on grounding and semantic categories; Scene Coherence is easiest in T2I because inconsistencies are visually obvious.Source: §4.3 · pp. 7–8, Figure 2 · p. 8

4.4 Does the choice of reference matter?

  • Changing references to paraphrased I2T answers and visually distinct but correct T2I images slightly lowers I2T performance but improves T2I performance.The pattern suggests text evaluators are sensitive to surface similarity, while image evaluators benefit from semantically correct visual diversity.Source: Table 4 · p. 8, §4.4 · p. 8

4.5 How does evaluator performance vary with reasoning budget?

  • Higher reasoning helps I2T Single Axes+Rubrics but not T2I, while low or medium reasoning generally outperforms higher budgets for Compare Axes+Rules.The authors note that the reasons remain speculative because reasoning traces were unavailable.Source: §4.5 · p. 8, Figure 3 · p. 9

4.7 Do justifications reveal failures beyond the scores?

  • Justification analysis provides only marginal improvement in perturbation detection, with a larger score-versus-justification gap for single-answer scoring than for pairwise comparison.Evaluators may recognize errors in single-answer justifications without penalizing them in their scores; pairwise judgments are better aligned with justifications.Source: Figure 4 · p. 9, §4.7 · p. 9

4.8 Practical recommendations for using VLMs as evaluators

  • Practitioners should favor structured pairwise evaluation, validate evaluator models on the target task, use moderate reasoning budgets, and supplement VLM judgments with human review or task-specific metrics for difficult errors.The recommendations especially caution against standalone VLM evaluation for fine-grained visual grounding, compositional reasoning, and physical concepts.Source: §4.8 · pp. 9–10

5 Related works

VLMs are increasingly used as evaluators for vision tasks, and prior work has identified biases in both LLM- and VLM-based evaluators. Existing VLM meta-evaluation has not systematically probed blind spots across paradigms; this work extends perturbation-based testing to I2T and T2I evaluation.

LLMs and VLMs as Auto Evaluators

Evaluation of Evaluators

6 Conclusion

FOCUS evaluates Evaluator VLM reliability across image-to-text and text-to-image tasks, revealing substantial blind spots and urging cautious, evidence-based deployment in benchmarking and training.

6 Conclusion

  • Evaluators miss substantial quality degradations, especially in fine-grained visual grounding, compositional reasoning, and physical plausibility; structured pairwise comparison is most reliable, while single-answer scoring is least dependable.The study also finds that increased reasoning budgets do not consistently improve reliability.Source: §6 · p. 10
  • FOCUS assesses four prominent VLMs across image-to-text and text-to-image tasks using targeted perturbations, human-in-the-loop validation, and three widely used evaluation paradigms.It is introduced as a meta-evaluation benchmark for Evaluator VLM reliability.Source: §6 · p. 10
  • Evaluators may recognize errors in their justifications without reflecting them in scores, and general-purpose model strength poorly predicts evaluation ability.These findings accompany the result that increased reasoning budgets do not consistently improve reliability.Source: §6 · p. 10
  • Evaluator blind spots may propagate when VLMs serve as training reward models, failing to penalize important errors; FOCUS aims to encourage cautious, evidence-based deployment.The authors present FOCUS as a diagnostic tool for benchmarking and training pipelines.Source: §6 · p. 10

Ethics Statement … A.2 Detailed descriptions of the perturbation categories

The paper reports its annotation, licensing, code-release, and language-assistance practices, and directs readers to appendix tables for detailed FOCUS statistics and perturbation descriptions.

Ethics Statement

  • Annotations were completed by proficient annotators paid competitive salaries in line with standard national wages.Source: Ethics Statement · p. 10
  • The datasets are available under permissible licenses, and the authors state they followed the licenses’ intended use and requirements.Source: Ethics Statement · p. 10
  • The authors state that evaluation and perturbation-generation code will be publicly released under the MIT License.Source: Ethics Statement · pp. 10–11
  • ChatGPT and similar assistants were used only to polish the authors’ original language, including paraphrasing and spell-checking, not to suggest new content.Source: Ethics Statement · p. 11

A.1 Detailed statistics of FOCUS

  • Table 5 presents detailed FOCUS statistics across all categories and perturbation dimensions.Source: §A.1 · p. 15

A.2 Detailed descriptions of the perturbation categories

  • Tables 7 and 8 provide detailed descriptions and examples of each perturbation type.Source: §A.2 · p. 15

A.3 Detailed descriptions of the benchmarks considered · B Human-in-the-Loop Validation of Perturbations

The benchmark appendix describes how I2T and T2I instances were sourced and how perturbation types were distributed. It also details a human-in-the-loop process for validating perturbations with a shared annotation interface and labeling rubric.

A.3 Detailed descriptions of the benchmarks considered

  • The benchmark draws 600 I2T instances from seven benchmarks and 750 T2I instances from seven benchmarks covering capabilities such as composition, counting, basic skills, and text rendering.I2T instances were manually selected from recent open-ended generation benchmarks; T2I instances were sampled across varied capabilities.Source: §A.3 · pp. 15–16
  • Reported I2T counts include entity substitution (105), attribute distortion (91), spatial relation swap (90), phantom details injection (83), and over-generalization (67).The same distribution also lists important detail omission (64), contextual depth reduction (43), cultural misalignment (76), and logical inconsistencies (59).Source: §A.3 · p. 16
  • Semantic-interpretation perturbation counts include numerical errors (86), procedural reordering (67), causal misattribution (58), ungrounded assumptions (55), misinterpreted key elements (62), and factual perturbations (60).Source: §A.3 · p. 16
  • Long-form generation includes narrative–visual conflict (76), thematic deviation (56), and tone-consistent mismatch (55), while score-neutral modifications number 473.Source: §A.3 · p. 16

B Human-in-the-Loop Validation of Perturbations

  • PerturbVal supports human validation for both tasks by showing the original input, gold output, and perturbed output, with word-level differences for I2T and side-by-side images plus edit instructions for T2I.Source: B Human-in-the-Loop Validation of… · p. 16, Figure 5 · p. 18, Figure 6 · p. 18
  • Annotators receive category guidelines, examples, and a brief description of each intended perturbation to support consistent interpretation and labeling.Source: B Human-in-the-Loop Validation of… · p. 17
  • Each instance receives one of five labels: Valid Perturbation, Score Invariant Perturbation, Incorrect Perturbation, Not Relevant, or Not Sure.Source: B Human-in-the-Loop Validation of… · p. 17
  • A perturbation is valid when it meaningfully degrades the gold output enough to warrant a lower evaluator score; preserved semantic correctness or unchanged evaluation outcomes qualify as score invariant.Incorrect labels cover perturbations that are misapplied, contradict their descriptions, or are absent from the output; unrelated and ambiguous cases are labeled Not Relevant and Not Sure, respectively.Source: B Human-in-the-Loop Validation of… · pp. 17–18

C Additional Evaluation details … Text-to-Image (T2I) Prompts

The appendix defines task-specific evaluation axes and supplies shared prompts for the four evaluator VLMs across tasks and strategies, alongside perturbation taxonomies for I2T and T2I. Axes-based evaluators return per-axis scores or verdicts, while non-axis strategies use the axes to shape their rubrics or rules.

C.1 Evaluation Axes

  • Axes-based strategies elicit per-axis scores or verdicts, while non-axis strategies use the task-specific axes to inform their rubrics or rules.Evaluators also produce an Overall Quality score in both settings.Source: §C.1 · p. 19, Text-to-Image (T2I) Axes · p. 20

Image-to-Text (I2T) Axes

  • I2T evaluation assesses relevance, trustworthiness, visual grounding, clarity, coherence, and completeness, with creativity assessed only in pairwise comparisons.Trustworthiness concerns accuracy and freedom from unsupported claims; visual grounding concerns reliance on evidence visible in the image.Source: Image-to-Text (I2T) Axes · p. 19

Text-to-Image (T2I) Axes

  • T2I evaluation covers prompt adherence, text-image alignment, image quality, accuracy, completeness, safety, and bias, alongside holistic Overall Quality.Bias evaluation checks whether occupations or education are represented without demographic stereotyping.Source: Text-to-Image (T2I) Axes · pp. 19–20

C.2 Prompts used for evaluation

  • Evaluation prompts are shared across all four evaluator VLMs for fair comparison and are organized by task and strategy.The appendix provides Vanilla, Axes, Rubrics/Rules, and combined Axes-plus-Rubrics or Axes-plus-Rules prompts for I2T and T2I.Source: §C.2 · p. 20

Image-to-Text (I2T) Prompts

  • I2T prompts cover Vanilla, Axes, Rubrics/Rules, Axes plus Rubrics, and Axes plus Rules strategies, with separate figures for each variant.Figures 7–11 show the listed single-answer, pairwise, or combined-strategy prompts.Source: Image-to-Text (I2T) Prompts · p. 20

Text-to-Image (T2I) Prompts

  • T2I prompts span five Vanilla, Axes, Rubrics/Rules, and combined-strategy variants; the accompanying taxonomy describes prompt violations and visual-fidelity or scene-coherence errors.The taxonomy includes object identity, count, attributes and spatial relations, as well as scale, behavior, incompleteness, missing context, mixed styles, anachronisms, incoherent layouts, and clutter.Source: Text-to-Image (T2I) Prompts · pp. 20, 23
  • I2T perturbations target plausible identity substitutions, subtle attribute changes, altered spatial relations, and fabricated but contextually consistent details.Other described perturbations broaden grounded terms, omit secondary details, strip contextual depth, substitute cultural references, or insert a subtle logical flaw.Source: Text-to-Image (T2I) Prompts · p. 21
  • Further I2T perturbations reduce mood and situational nuance, misrepresent cultural identity, or bury a logical inconsistency in an otherwise coherent description.These perturbations are designed to require scrutiny beyond surface-level factual accuracy or first-pass reading.Source: Text-to-Image (T2I) Prompts · p. 21

D Detailed Examples for I2T tasks … D.3 Visual Reasoning

The examples show how plausible I2T answer perturbations can alter object identity, attributes, spatial relations, visual details, and narrative interpretation. They contrast each altered description with a gold answer, illustrating changes that require close visual inspection or reasoning to notice.

D.1.1 Entity Substitution

  • The examples substitute raspberries for strawberries and chocolate for rainbow sprinkles, preserving plausible descriptions while changing fruit identity and cupcake toppings.The fruit example distinguishes berries inside baskets from those loose on the table; the cupcake example changes the toppings on pink-frosted cupcakes.Source: §D.1.1 · p. 25, Perturbed · p. 25, Gold · p. 25

D.1.3 Spatial Relation Swap

  • The tiger cutout is changed from protruding into the aisle and occluding yellow cereal boxes to recessed behind boxes and itself partially occluded.The perturbation reverses the cutout’s depth and visibility relative to the shelf and aisle.Source: §D.1.3 · p. 26, Gold · p. 26, Perturbed · p. 26

D.1.4 Phantom Details Injection

  • The street-scene perturbation fabricates a glowing pedestrian Walk signal on the foreground lamppost and a red shopping bag carried by a pedestrian.The rationale describes both additions as contextually plausible, while the gold answer identifies the lamppost as the closest object and pedestrians farther back.Source: Perturbed · p. 26, Gold · p. 26

D.1.5 Over Generalization

  • The cat description replaces specific terms such as “orange tabby” and “hardcover books” with broader labels while retaining the account of its occluded body.The gold answer specifies that a tall horizontal stack of thick books blocks the cat’s lower face, torso, and legs; the perturbed version generalizes these details.Source: §D.1.5 · p. 27, Gold · p. 27, Perturbed · p. 27

D.1.6 Important Detail Omission

  • The pasture perturbation omits the counts of 12 sheep and 3 alpacas and the Border Collie’s action, while retaining a fluent, visually grounded scene description.The gold account also identifies one Border Collie; the perturbed passage retains the dog’s presence and location but not its action.Source: Perturbed · p. 27, Gold · p. 27

D.2.1 Contextual Depth Reduction

  • The kitchen perturbation removes the gold answer’s causal narrative about the dog using a stool to reach the counter and retreating in guilt, leaving a sequence of scene details.The rationale says it strips narrative causality, emotional resonance, and inferential depth; the gold account links the stool, spill, and dog’s retreat.Source: §D.2.1 · p. 28, Gold · p. 28, Perturbed · p. 28

D.2.2 Cultural Misalignment

  • The driveway perturbation replaces “snowman,” “carrot nose,” and “scarf” with “yuki-daruma,” a wooden kokeshi nose, and a tenugui in an otherwise coherent narrative.The rationale notes that the cultural substitutions are difficult to detect from text alone because the account remains grammatically correct and contextually plausible.Source: Perturbed · p. 28, Gold · p. 28

D.2.3 Logical Inconsistencies

  • The bystander perturbation changes defensive behavior such as leaning away and crossing arms to leaning forward, a plausible action in an office setting.The gold passage describes the bystanders’ avoidance and fidgeting as signaling unease and reluctance to engage.Source: §D.2.3 · p. 29, Gold · p. 29

D.3.1 Numerical Errors … D.4 Long-form Generation

The examples show how targeted perturbations can preserve plausible reasoning while altering numerical, visual, causal, factual, or thematic content. They contrast gold answers with perturbed versions but provide no evaluator performance results.

D.3.1 Numerical Errors

  • The numerical perturbation changes the active-commuting goal from 25% to 20% and recalculates the target and remaining commuters accordingly.The gold answer uses a 25% increase of 20,000 to reach 25,000 active commuters and 75,000 remaining; the perturbed answer uses 20% to reach 24,000 and 76,000 remaining.Source: Perturbed · p. 29, Gold · p. 29

D.3.2 Procedural Reordering

  • The procedural-reordering example swaps the cat’s and dog’s visual evidence while retaining a plausible deduction that implicates the dog in the spill.The gold chain attributes stained paws to the cat; the perturbation instead attributes them to the dog and gives the cat clean paws.Source: §D.3.2 · p. 30, Gold · p. 30, Perturbed · p. 30

D.3.3 Causal Misattribution

  • The causal-misattribution example reverses the accident sequence, claiming the sedan accelerated first rather than responding to the truck losing traction.The gold description says the truck jackknifes and blocks the uphill lane while the sedan slides out of control; the perturbation attributes the sedan’s state to aggressive acceleration.Source: Perturbed · p. 30, Gold · p. 30

D.3.4 Ungrounded Assumptions

  • The ungrounded-assumptions perturbation adds definitive claims that the child was startled, will cry, and will halt a birthday party.The gold and perturbed descriptions both include the balloon being released and heading toward a thorny bush; the added emotional and social conclusions are identified as unverified.Source: §D.3.4 · p. 31, Gold · p. 31, Perturbed · p. 31

D.3.5 Misinterpret Key Elements

  • The train-board perturbation misaligns status labels with train details, changing the 10:45 AM Washington Express from canceled to delayed.The rationale says the perturbed answer associates “DELAYED 45 MIN” with the 10:45 train and “CANCELED” with the 11:00 train; the gold response sends the passenger to Track 5 for the 11:00 AM train.Source: Perturbed · p. 31, Gold · p. 31

D.3.6 Factual Perturbations

  • The factual perturbation assigns the man’s Chicago destination to Miami-bound Train 810 while retaining correct time arithmetic for a 30-minute wait.The gold answer identifies Chicago-bound Train 505 at Track 4, departing at 10:50 AM, with five minutes remaining at 10:45 AM.Source: §D.3.6 · p. 32, Gold · p. 32

D.4 Long-form Generation

  • The long-form thematic perturbation replaces the requested storm-as-attacker and lighthouse-as-defender framing with an ocean described as timeless and shaping.The passage identifies this as a thematic shift in the requested personification.Source: §D.4.2 · p. 33

E Detailed examples for T2I tasks … E.4.4 Mislabeled Symbols/Diagrams

This section presents T2I perturbation examples that violate explicit prompt requirements while preserving much of each scene, across object, scene, physical, and text-related constraints. It also shows evaluator prompts for mislabeled symbols and diagrams, including compare-axes-plus-rules and reference prompts.

E.1.1 Object Substitution; E.1.2 Object Addition/ Omission; E.1.3 Attribute Manipulation

  • Object edits replace the requested human astronaut with a visually similar robot, add a second banana, or make a matte vase glossy while preserving other scene attributes.These changes violate subject identity, singular count, and finish requirements, respectively.Source: §E.1.1 · p. 34, §E.1.2 · p. 34, §E.1.3 · p. 35

E.1.4 Spatial Manipulation; E.1.5 Scale Distortion; E.1.6 Constraint Violation

  • Spatial and scale edits move a knife to the wrong side of a basket or shrink a concrete bucket, while another edit has a miniature knight stand beside rather than ride its horse.The objects and much of the requested scene remain present, obscuring violations of position, scale, or action constraints.Source: §E.1.4 · p. 35, §E.1.5 · p. 36, §E.1.6 · p. 36

E.2.1 Incomplete Scene; E.2.2 Missing Context; E.2.3 Style Inconsistency

  • Scene edits render a shell harp as a rough sketch, remove a required candle chandelier, or turn a stone bridge into a cartoon, violating completeness, context, or stylistic consistency.The missing chandelier is especially easy to overlook because other scene elements still provide illumination.Source: §E.2.1 · p. 37, §E.2.2 · p. 37, §E.2.3 · p. 38

E.2.4 Theme Conflict; E.2.5 Disorganized Composition; E.2.6 Overcrowding

  • Other scene edits add a modern multimeter to a Victorian laboratory, separate a chef’s hat from a dinosaur, or clutter a serene scholarly gathering.These violate period, wearing-relation, and atmosphere requirements; the added multimeter fits the broad laboratory theme, while clutter objects fit calligraphy.Source: §E.2.4 · p. 38, §E.2.5 · p. 39, §E.2.6 · p. 39

E.3.1 Causal Violation; E.3.2 Physics Manipulation; E.3.3 State/Transformation Failure

  • Physical-reasoning edits flatten a trampoline beneath a bowling ball, reverse a prism’s light dispersion, or remove refraction’s checkerboard distortion inside a glass sphere.The requested objects remain recognizable, but their causal, optical, or transformed-state properties are wrong.Source: §E.3.1 · p. 40, §E.3.2 · p. 40, §E.3.3 · p. 41

E.3.4 Functional Absurdity; E.3.5 Literalized Idioms

  • Further physics examples replace a prism’s rainbow with white light or give a sphere’s shadow a pointy blade-like tip, despite otherwise plausible scenes.The edits violate the requested optical function and the natural shape of a sphere’s shadow.Source: §E.3.4 · p. 41, §E.3.5 · p. 42

E.4.1 Text/Typographical Corruption; E.4.2 Incomplete Rendering; E.4.3 Background Misrendering

  • Text perturbations misspell “Breathe” and “RESERVED,” erase “OUT” from “KEEP OUT,” or leave “BEAR LAKE 2 MILES” floating after removing its signboard.These can evade notice when evaluation recognizes text shape or scene context without checking exact wording or its physical support.Source: §E.4.1 · pp. 42–43, §E.4.2 · p. 43, §E.4.3 · p. 44

E.4.4 Mislabeled Symbols/Diagrams

  • A final text edit changes the jacket patch from “WILDERNESS EXPLORER” to “WILDLIFE EXPLORER,” retaining a similar prefix and semantic theme.The change violates the prompt’s exact-text requirement while keeping the image’s overall context intact.Source: §E.4.4 · p. 44
  • The subsection shows T2I evaluator prompts for comparing axes plus rules and for reference evaluation.The examples are identified as Figure 17 and Figure 18, respectively.Source: Figure 17 · p. 55, Figure 18 · p. 56

Related papers

Loading 2604.21523v1…