PaperlayerView exact revision on arXiv

Source-linked AI summary

Seeing Isn't Believing: Uncovering Blind Spots in Evaluator Vision-Language Models

Mohammed Safi Ur Rahman Khan, Sanjay Suryanarayanan, Tushar Anand, Mitesh M. Khapra

arXiv:2604.21523v1cs.CVcs.CL

TL;DR

Evaluator VLMs are increasingly used to judge model outputs, but their reliability across fine-grained I2T and T2I errors remains underexplored. FOCUS tests this reliability with targeted, human-validated perturbations across multiple evaluation paradigms and finds substantial blind spots, making standalone reliance unsafe.

  • Problem

    Evaluator VLMs increasingly shape rankings and training feedback, yet evidence is limited about whether they reliably detect fine-grained errors in I2T and T2I outputs.

  • Method

    FOCUS evaluates four VLMs on over 4000 human-validated perturbations across I2T and T2I using single-answer, pairwise, and reference-guided paradigms.

  • Results

    Pairwise comparison is the most reliable paradigm, but evaluators still frequently miss degradations, with some failure rates exceeding 50%.

  • Takeaways & Limitations

    Evaluator VLMs exhibit substantial blind spots, so benchmarking and training decisions should not rely on them as standalone judges.

  • Takeaways & Limitations

    The study notes that gold outputs need only be reasonably accurate because it evaluates directional score changes rather than perfect gold quality.

Abstract

from arXiv · show

Large Vision-Language Models (VLMs) are increasingly used to evaluate outputs of other models, for image-to-text (I2T) tasks such as visual question answering, and text-to-image (T2I) generation tasks. Despite this growing reliance, the reliability of these Evaluator VLMs remains under explored. In this work, we systematically evaluate the reliability of Evaluator VLMs across both I2T and T2I tasks. We introduce targeted perturbations that degrade output quality along key error dimensions, including object hallucinations, spatial reasoning, factual grounding, and visual fidelity. These perturbations test whether Evaluator VLMs can reliably account for these quality degrading errors in their evaluations. Using a comprehensive benchmark of over 4000 perturbed instances spanning 40 perturbation dimensions, we evaluate 4 prominent VLMs using single-answer scoring, pairwise comparison, and reference-guided paradigms. Our findings reveal that current VLM evaluators exhibit substantial blind spots: they often fail to detect perturbed outputs - in some cases exceeding 50%, struggle particularly with fine-grained compositional and spatial errors, and are often insensitive to hallucinated content that contradicts the input image. Pairwise comparison proves more reliable, though failure rates persist. These results highlight the unreliable nature of current Evaluator VLMs and urge caution in their deployment for benchmarking and development decisions. Code and data have been made publicly available.

1 Introduction

FOCUS addresses whether Evaluator VLMs reliably detect quality-degrading errors across image-to-text and text-to-image evaluation. The benchmark reveals substantial blind spots, with failures especially common for fine-grained visual and compositional judgments.

  • Motivation: Evaluator VLMs increasingly assess model outputs because they are scalable and cost-effective, but unreliable judgments can mislead rankings and reward undesirable behaviors during training.This motivates rigorous reliability assessment beyond correlation with human judgments.
  • Approach: FOCUS introduces targeted perturbations across diverse failure modes to test whether evaluators detect clear degradations in I2T and T2I outputs.The benchmark contains over 4000 human-validated perturbed instances generated from real-world prompts and gold responses.
  • Evaluation: Four prominent VLMs are evaluated using single-answer scoring, pairwise comparison, and reference-guided evaluation.The study examines prompting strategies used in existing literature and practice.
  • Findings: Failure rates can exceed 50%, with evaluators performing worse on T2I and struggling with fine-grained visual grounding, compositional reasoning, and physical plausibility.Evaluators sometimes identify errors in justifications without reflecting them in final scores.
  • Findings: Pairwise comparison is the most reliable paradigm, although current evaluators still show substantial failures in detecting quality-degrading perturbations.The findings caution against relying on Evaluator VLMs as standalone judges.
  • Implications: Blind spots in evaluator VLMs may allow reward signals during training to reinforce behaviors they should correct.The paper highlights implications for both benchmarking and model development.

2 FOCUS benchmark

FOCUS is a two-split meta-evaluation benchmark that tests evaluator reliability by comparing gold outputs with human-validated perturbations. Its perturbations cover diverse I2T and T2I error dimensions and are generated automatically before thorough human review.

  • Benchmark structure: FOCUS contains I2T and T2I splits for evaluating how well VLMs judge outputs from other VLMs and image generation models.I2T includes tasks such as VQA and captioning, while T2I covers text-to-image generation.
  • Benchmark structure: Each instance pairs a gold output with a perturbed output containing controlled errors, enabling evaluation of whether judges lower scores for degraded results.Human oversight covers prompt selection, perturbation definitions, and perturbation creation.
  • Data construction: The benchmark samples 600 I2T instances and 750 T2I instances from recent evaluation benchmarks.Gold answers and images are generated with GEMINI-3.1-PRO and GEMINI-3-PRO-IMAGE, respectively, then manually reviewed.
  • I2T perturbations: I2T perturbations cover text rendering, visual grounding, semantic interpretation, visual reasoning, and long-form generation.Examples include altered object identities, contextual cues, numerical claims, and inconsistencies between text and imagery.
  • T2I perturbations: T2I perturbations cover visual fidelity, scene coherence, and physical plausibility, including altered attributes, stylistic mismatches, and violations of physical laws.One example is rendering a red car as a blue car; another is producing shadows that point toward the light source.
  • Validation: Perturbations are generated through an automatic pipeline and then reviewed by annotators who label them according to whether they introduce meaningful score-degrading errors.The process distinguishes valid, invalid, and score-invariant perturbations.

3 Experimental Setup

The study compares three evaluator paradigms and four increasingly structured prompting strategies across I2T and T2I tasks. It measures whether evaluators detect perturbations through unchanged scores, incorrect pairwise choices, or perfect scores on degraded outputs.

  • Evaluation paradigms: The evaluation paradigms are single-answer scoring, pairwise comparison, and reference-guided scoring.Evaluators receive task inputs and candidate outputs, with reference-guided scoring additionally providing a gold output.
  • Prompting strategies: Prompting strategies progressively add structure through vanilla inputs, rubrics or rules, predefined axes, and axes combined with rubrics or rules.Axes-based evaluators produce per-axis judgments where applicable.
  • Evaluation paradigms: Single-answer scoring evaluates one response independently, whereas pairwise comparison selects the better of two candidates.Pairwise inputs contain two answers for I2T or two images for T2I.
  • Evaluation paradigms: Reference-guided scoring compares a model output against an explicit gold answer or image, which may simplify judgment but is not feasible when good references are unavailable.The reference is Agold for I2T and Igold for T2I.
  • Metrics: The metrics quantify undetected perturbations as unchanged scores, failures to select the gold output, or perfect scores assigned to perturbed outputs.For pairwise evaluation, output order is swapped to mitigate position bias.

4 Results and discussion

Across tasks and evaluator models, structured pairwise comparison is generally the most reliable paradigm, while single-answer scoring remains weakest. Reliability nevertheless varies by task, model, prompting strategy, perturbation category, reference choice, reasoning budget, and whether justifications are considered.

  • Paradigms and strategies: Pairwise comparison is the most reliable paradigm across I2T and T2I, while single-answer scoring is the weakest and can exceed 50% failure in T2I.Reference-guided evaluation improves over single-answer scoring but generally remains behind the best pairwise strategies.
  • Paradigms and strategies: Structured Axes and Axes+Rules strategies consistently improve reliability, whereas generic rubrics alone can degrade single-answer performance.The strongest gains appear for pairwise comparison, particularly in T2I.
  • Evaluator models: GEMINI-3.1-PRO has the lowest pairwise failure rates across I2T and T2I, while QWEN3.5-397B-A17B tends to fail more often, especially on T2I.CLAUDE-OPUS-4.6 is competitive under structured I2T strategies but otherwise shows relatively higher failure rates; GPT-5.4 remains slightly behind GEMINI-3.1-PRO.
  • Perturbation categories: Visual Grounding and Semantic Interpretation are hardest in I2T single scoring, while Physical Plausibility is hardest in T2I and Scene Coherence is easiest.Pairwise comparison substantially reduces failures on the challenging I2T categories, whereas physical-plausibility errors require deeper reasoning about physics or common sense.
  • Reference variation: Reference variation slightly worsens I2T performance but improves T2I performance, indicating different sensitivity to textual similarity and visually diverse references.The default reference is replaced with a paraphrased I2T answer or a visually distinct but correct T2I image.
  • Reasoning budget: Increasing reasoning does not consistently improve reliability: high reasoning helps I2T single scoring but harms T2I single scoring and comparative evaluation.For Compare Axes+Rules, low or medium reasoning generally performs better across both tasks, while high reasoning can produce high failure rates in T2I single scoring.
  • Justifications and score-invariant perturbations: For score-invariant perturbations, single-answer scoring is most robust and pairwise comparison least stable, while reference-guided scoring offers an intermediate alternative.Pairwise evaluators may prefer one of two equally good candidates, whereas references can anchor judgments but penalize valid alternatives that differ from them.
  • Justifications and score-invariant perturbations: Justifications provide only marginal detection gains, because evaluators may recognize errors without appropriately penalizing them in final scores.The gap between justification-aware and score-only detection is more pronounced for single-answer scoring than pairwise comparison.

5 Related works

Prior work uses language and vision-language models as automatic evaluators, while evaluating evaluator reliability through human alignment studies and robustness testing. Existing VLM research covers image generation and VQA evaluation, but this paper targets systematic blind spots.

  • LLMs and VLMs are increasingly used as automatic evaluators for text, VQA, and text-to-image generation outputs.VLM evaluators operate through single-answer scoring, pairwise comparison, and reward-model settings.
  • Evaluator reliability has been studied through correlations with human judgments and robustness testing with adversarial perturbations.These approaches span text-based evaluator LLMs and more recent VLM evaluation studies.
  • Recent VLM work examines alignment with human preferences in image generation and reliability in VQA evaluation.

6 Conclusion

The paper introduces FOCUS to assess Evaluator VLM reliability across image-to-text and text-to-image tasks. It finds substantial blind spots, with structured pairwise comparison most reliable and single-answer scoring least dependable.

  • FOCUS evaluates four prominent VLMs across I2T and T2I using targeted perturbations and human-in-the-loop validation.
  • Evaluator VLMs frequently miss quality degradations, especially those requiring fine-grained visual grounding, compositional reasoning, or physical plausibility.
  • Structured pairwise comparison is the most reliable paradigm, whereas single-answer scoring is the least dependable.
  • Increased reasoning budgets do not consistently improve reliability, and evaluators may recognize errors in justifications without reflecting them in scores.
  • Evaluator blind spots may propagate into reward-model training and fail to penalize important errors.

Ethics Statement

The paper reports ethical and reproducibility practices concerning annotation, dataset licensing, public code release, and use of language assistants. The supplied passages also reference detailed benchmark statistics and perturbation descriptions.

  • FOCUS annotations were produced by proficient annotators paid competitive salaries consistent with standard national wages.
  • The datasets were used under permissible licenses and according to their intended usage.
  • The evaluation and perturbation-generation code was planned for public release under the MIT License.
  • The paper used ChatGPT and similar assistants only for language assistance such as paraphrasing, spell-checking, and polishing.
  • Tables 5, 7, and 8 provide detailed benchmark statistics and perturbation descriptions with examples.

A.3 Detailed descriptions of the benchmarks considered

FOCUS constructs I2T and T2I perturbation benchmarks from multiple open-ended-generation datasets and validates perturbations through a dedicated human annotation interface. The benchmark organizes errors into fine-grained dimensions and labels their validity and evaluation impact.

  • Benchmark sources: The I2T split manually selects 600 instances from seven popular benchmarks focused on open-ended generation.
  • Perturbation taxonomy: I2T perturbations span semantic interpretation, long-form generation, and score-invariant modifications across dimensions such as entity, attribute, spatial, factual, and narrative errors.
  • Perturbation taxonomy: FOCUS includes score-neutral modifications as score-invariant perturbations.
  • Benchmark sources: The T2I split samples 750 instances from seven benchmarks covering composition, counting, basic skills, and text rendering.
  • Human validation: PerturbVal shows annotators the original input, gold output, and perturbed output, using word-level differences for I2T and side-by-side images for T2I.
  • Human validation: Annotation guidelines describe perturbation categories, expected behaviors, examples, and labeling instructions to support consistency.
  • Human validation: Each instance receives one of five labels: Valid Perturbation, Score Invariant Perturbation, Incorrect Perturbation, Not Relevant, or Not Sure.
  • Human validation: A valid perturbation meaningfully degrades the output so that a reliable evaluator should assign a lower score.

C.1 Evaluation Axes

The benchmark defines task-specific evaluation axes and uses targeted perturbations to test whether evaluators detect subtle errors in I2T and T2I outputs. It evaluates multiple prompting strategies across single-answer and pairwise paradigms.

  • Evaluation Axes: Task-specific axes guide evaluator judgments for both I2T and T2I settings.Axis-based strategies produce per-axis scores or verdicts, while non-axis strategies use the definitions in rubrics or rules.
  • Evaluation Axes: I2T evaluation includes relevance, trustworthiness, and visual grounding axes.Trustworthiness covers accuracy, image grounding, and hallucinated or unsupported claims.
  • Evaluation Axes: T2I evaluation includes prompt adherence, text-image alignment, image quality, and bias.These axes cover objects, attributes, actions, spatial relationships, visual artifacts, and demographic stereotyping.
  • Evaluation Axes: Overall Quality provides a holistic assessment across the task-specific axes.The score evaluates general effectiveness, faithfulness, and suitability of the generated output.
  • Prompting Strategies: The benchmark tests vanilla, axes, rubric or rule, and combined prompting strategies in single-answer and pairwise evaluation.The prompts are shared across all four evaluator VLMs for fair comparison.
  • Perturbation Dimensions: Perturbations target identity, object count, attributes, spatial relations, scale, hallucinated details, omissions, contextual depth, cultural alignment, and logical consistency.These modifications preserve plausibility while introducing errors requiring explicit comparison with the source or prompt.

D Detailed Examples for I2T tasks

The I2T examples demonstrate perturbations that preserve fluent, plausible descriptions while changing fine-grained factual, spatial, numerical, causal, or cultural information. These cases require evaluators to verify details against the image rather than rely on contextual plausibility.

  • Fine-Grained Factual Errors: Replacing strawberries with raspberries preserves a plausible fruit description while changing the depicted fruit identity and count.Both are small red berries, making the substitution difficult to detect from textual coherence alone.
  • Fine-Grained Factual Errors: Changing rainbow sprinkles to chocolate sprinkles tests whether evaluators inspect fine visual attributes on cupcakes.The altered description remains fluent and contextually plausible.
  • Spatial Errors: Reversing a cardboard tiger’s depth relationship changes it from protruding and occluding cereal boxes to being recessed behind them.The two descriptions maintain a physically possible shelf arrangement while contradicting the image’s spatial layout.
  • Hallucinated Details: Injecting a pedestrian signal and shopping bag adds plausible details that are absent from a rainy urban street image.The additions are designed to test whether evaluators detect hallucinated content that blends with genuine observations.
  • Descriptive Detail: Replacing specific terms such as “orange tabby cat” and “thick hardcover books” with hypernyms reduces descriptive precision while preserving surface accuracy.The perturbation tests whether evaluators notice loss of identifying detail and grounding specificity.
  • Omissions: Removing exact animal counts and the Border Collie’s action produces a fluent description that omits explicit numerical and relational information.The altered answer remains broadly factually accurate while losing requested detail.
  • Cultural Misalignment: Cultural substitution replaces Western snowman terminology with Japanese equivalents while keeping the narrative coherent and contextually plausible.The example tests whether evaluators detect culturally specific misalignment rather than only grammatical or logical defects.

D.3.1 Numerical Errors

The numerical-error examples show that I2T perturbations can preserve fluent reasoning while changing numerical premises, entity-to-evidence assignments, causal direction, or unsupported inferences. Detecting these errors requires checking each claim against the visual evidence and prompt constraints.

  • Numerical Errors: Changing an active-commuting target from 25% to 20% changes the computed non-active commuter count from 75,000 to 76,000.The perturbed calculation remains arithmetically consistent with its altered premise.
  • Evidence Attribution: Swapping paw evidence between a cat and dog reverses which animal is identified as responsible for the spilled mug.The perturbed answer preserves a plausible deductive chain but assigns the visual evidence to the wrong entity.
  • Causal Reasoning: Reversing the accident’s cause-and-effect sequence changes the explanation from truck-induced loss of traction to sedan-induced acceleration.The error introduces a plausible alternative account of the vehicles’ visual states.
  • Unsupported Inference: Adding definitive claims about a child’s emotional reaction and a birthday party extends beyond the observable balloon trajectory.The perturbation tests unsupported conclusions about internal state and social context.
  • Structured Information: Misaligning train statuses with departure information changes the recommended track and train choice despite preserving the timetable’s surface details.The perturbation targets correspondence between entries in different parts of a visual schedule.
  • Numerical Errors: Associating Chicago with Miami-bound train details yields a fluent 30-minute calculation but the wrong destination and departure information.The example separates arithmetic correctness from correct entity-to-attribute alignment.

E Detailed examples for T2I tasks

The T2I examples target errors that preserve overall scene plausibility while violating subject identity, object count, visual attributes, spatial relations, scale, actions, style, or historical consistency. Their rationales emphasize why coarse semantic matching can miss these violations.

  • Identity: Replacing a human astronaut with a humanoid robot preserves pose and science-fiction appearance while violating subject identity.The shared silhouette and aesthetic make the substitution easy to overlook.
  • Object Count: Adding a second banana violates the prompt’s singular constraint while producing a natural-looking countertop scene.The added object matches the existing object’s semantic context and visual style.
  • Visual Attributes: Changing a vase from matte to glossy preserves its identity, color, and placement while violating its specified finish.The example isolates a fine-grained material attribute.
  • Spatial Relations: Moving a knife from right to left of a basket violates an explicit spatial relation while retaining all requested objects.Object presence and surface placement remain correct, so detection requires checking relative position.
  • Scale: Shrinking a heavy metal bucket to soup-can size violates the requested scale while preserving the pouring action.The object remains technically a metal bucket and the broader scene semantics remain intact.
  • Object Omission: Removing a floating-candle chandelier violates an explicit scene requirement but can remain visually plausible because other elements provide illumination.The spellbook, dragon, and stained-glass windows mask the missing lighting source.
  • Contextual Consistency: Introducing a modern multimeter into a Victorian laboratory creates a historical anachronism that fits the broad laboratory context.Semantic compatibility can obscure the period-specific violation.
  • Relational Errors: Moving a chef’s hat from the dinosaur’s head to the table changes a wearing relation while retaining both entities in the scene.The example targets verification of a specific object-to-subject relation rather than object presence.
Loading 2604.21523v1…