Source-linked AI summary
Blind-Spots-Bench: Evaluating Blind Spots in Multimodal Models
Matteo Santelmo, Xiuying Wei, Israa Fakih, Felix Bauer, Juan Garcia Giraldo, Chengkun Li, Etienne Bamas, Emmanuel Abbé
TL;DR
Humans find many simple tasks easy, yet modern models still exhibit persistent blind spots that existing benchmarks may under-measure. Blind-Spots-Bench evaluates these weaknesses across 235 curated problems and diverse models, finding that closed-source frontier systems outperform open-weight models by approximately 10% while no model dominates every task type.
Problem
Modern models still fail on tasks straightforward for humans, revealing persistent blind spots across spatial reasoning, logical consistency, and character-level manipulation.
Method
The paper introduces a 235-question benchmark with structured reference solutions, task taxonomy, and an automated solver-grader pipeline for evaluating diverse AI models.
Results
Closed-source frontier systems achieve approximately 10% higher accuracy than the highest-scoring open-weight models, while no single model dominates across all task types.
Takeaways & Limitations
Blind-Spots-Bench provides a diagnostic stress test for comparing model weaknesses and identifying shared and model-specific failure patterns.
Takeaways & Limitations
The dataset is modest and imbalanced, may be biased toward weaknesses of two frontier models, and lacks a human baseline.
Abstract
from arXiv · showhide
Modern AI models achieve strong performance on many established benchmarks, yet they still fail on tasks that humans find almost trivial, such as manipulating a string or drawing a dog with five legs. These examples suggest that existing benchmarks may under-measure persistent blind spots in current systems. We introduce $\texttt{blind-spots-bench}$, a benchmark designed to expose such blind spots through tasks that appear simple for humans but remain challenging for modern AI. We collect raw questions from students in an AI course, clean and annotate them with structured reference solutions, and propose a task taxonomy tailored to the resulting dataset of 235 samples. We further develop an automated grading pipeline to evaluate a wide range of models, including open-weight and closed-source language, vision-language, and image-generation models. Our analysis on $\texttt{blind-spots-bench}$ reveals that closed-source frontier models can substantially outperform open-weight models with even $\approx10\%$ gap, even when they attain comparable performance on existing benchmarks. A more fine-grained analysis shows that no single model dominates across all task types, and that some tasks remain challenging for all evaluated models. These results highlight the value of $\texttt{blind-spots-bench}$ as a diagnostic stress test for identifying concrete weaknesses in current modern models.
1 Introduction
Despite strong performance and near-saturation of established benchmarks, modern language and multimodal models still show persistent blind spots on tasks humans find straightforward. Blind-Spots-Bench addresses this gap with 235 curated questions, structured evaluation, and analyses of model performance across task types.
- Motivation: Modern language and multimodal models achieve strong results across mathematics, coding, reasoning, vision, and vision-language benchmarks, sometimes rivaling or surpassing human experts.
- Motivation: Models nevertheless fail on human-trivial tasks involving exact string lengths, object counts, clock times, or simple Sudoku, revealing blind spots in spatial reasoning, logical consistency, and character handling.
- Benchmark: Blind-Spots-Bench contains 235 manually curated questions collected from graduate AI students and annotated with structured reference solutions for reproducible evaluation.
- Results: Approximately 10 per cent higher accuracy separates the best closed-source systems from the highest-scoring open-weight models, even when established-benchmark performance is comparable.
- Evaluation: The evaluation pipeline covers response generation, AI-based grading, and manual reliability audits across 32 LLMs/VLMs and 6 specialized image-generation models.
- Analysis: A taxonomy of three high-level categories and 12 subcategories supports fine-grained analysis, revealing shared weaknesses such as object counting alongside model-specific strengths and failure patterns.
2 Related Works
Prior work increasingly moves beyond aggregate benchmark scores to characterize model behavior, reasoning errors, and robustness through structured evaluations. These efforts include broad reasoning benchmarks, diagnostic taxonomies, failure analyses, and perturbation-based robustness tests that complement blind-spots-bench.
- General and Multimodal Evaluation Benchmarks: Broad benchmarks such as RBench, BIG-Bench, GPQA, and MMMU aggregate multidisciplinary reasoning metrics but do not distinguish behavior across specific reasoning components.This motivates more structured characterizations of model reasoning errors beyond a single aggregate score.
- Taxonomies and Failure Analysis: Prior diagnostic work uses behavioral testing, challenge sets, task taxonomies, and failure analysis to reveal brittle heuristics despite high benchmark accuracy.Examples include checklist in NLP and HANS as a diagnostic benchmark.
- Robustness Evaluation: Robustness benchmarks test reasoning invariance by generating perturbed problems or parametric templates, complementing blind-spots-bench while serving a different purpose.GSM-Plus and Math-Perturb generate controlled variants, whereas GSM-Symbolic and VarBench test equivalent problems through parametric templates.
3 Benchmark construction and Composition analysis
Blind-spots-bench was constructed from student-proposed questions through cleaning, structured annotation, review, and difficulty filtering. Its composition spans three input-output formats and a unified taxonomy whose frequent subtasks reflect known model weaknesses.
- Benchmark construction: Approximately 287 raw student questions were standardized by filtering difficult or duplicate entries and annotating them with structured reference solutions.Reference solutions specify expected answers, correctness conditions, and common failure modes for automatic verification.
- Question formats: The dataset includes text-only, image-generation, and multi-to-text question formats to support comparisons among models with different input-output capabilities.Text-only uses text inputs and outputs; image-generation produces images; multi-to-text combines image and text inputs with text output.
- Task taxonomy: The taxonomy groups tasks into Object-centric, Abstract reasoning, and Language and knowledge categories tailored to the benchmark’s diverse skills.It is intended to expose reasoning blind spots and support analysis of model performance across question types.
- Composition analysis: Text-only questions comprise 46.2% of the dataset, followed by image-gen at 35.6% and multi-to-text at 18.2%.These proportions describe the question-format composition reported in Fig. 3.
- Composition analysis: Spatial reasoning appears 53 times, while perceptual counting, generative counting, and logical reasoning are also frequent subtasks linked to recognized model difficulties.The analysis connects these frequencies to struggles with spatial relations, visual transformations, precise numerical constraints, novel-quantity binding, and multi-step inference.
- Task taxonomy: The taxonomy unifies individual subtasks into a dataset-specific framework connecting composition, known model weaknesses, and subsequent performance analysis.This organization is presented as a contribution beyond studying the subtasks separately.
4 Experiments
Experiments evaluate diverse language, vision-language, and image-generation models using an automated solver–grader pipeline. Results show strong but uneven performance, cost–accuracy trade-offs, persistent visual bottlenecks, complementary model strengths, and inconsistent scaling benefits.
- Evaluation Pipeline: The evaluation pipeline prompts solver models without examples or chain-of-thought, then uses a second grader model to produce binary correctness judgments.The pipeline is designed to automate evaluation at scale across blind-spots-bench.
- Evaluated Models Choice and Configuration: The benchmark covers state-of-the-art LLMs, VLMs, and image-generation models, with thinking enabled when available, medium effort, and up to 32,768 output tokens.Image-generation evaluation includes GPT-Image and Gemini-Image models, but no open-weight image-generation models because of technical constraints.
- Overall Performance: Gemini-3.1-Pro reaches ≈83.3% textual and ≈66.9% multimodal accuracy, while GPT-5.5 reaches ≈84% textual but ≈58.7% visual accuracy.Strong open-weight models remain mainly competitive on text-only problems and lag in multimodal settings.
- Overall Performance: 54.8% is Gemini-3-Pro-Image’s accuracy, compared with 51.2% for GPT-Image-2, which is approximately 4× cheaper.Gemini-3-Pro-Image attains the highest image-generation accuracy but at substantially higher inference cost.
- Fine-grained Analysis: Fine-grained visual perception remains a bottleneck: the strongest models achieve only 41.67% on attribute and pattern recognition and 57.14% on perceptual counting.Models also show complementary strengths, larger variants sometimes underperform smaller ones, and open models trail closed models with comparable AAII scores.
5 Conclusion
The paper introduces blind-spots-bench as a 235-question benchmark with a fine-grained taxonomy, curated reference solutions, and automatic grading for diverse models. It finds closed-source frontier models outperform open-weight models, while noting limitations from dataset size, subtask imbalance, and potential model-specific bias.
- Contributions: blind-spots-bench contains 235 questions that challenge modern models while remaining straightforward for humans.The benchmark targets persistent model failure modes despite apparent task simplicity.
- Contributions: The benchmark provides a fine-grained taxonomy, curated reference solutions, and an automatic grading pipeline for diverse VLMs, LLMs, and image-generation models.These components support structured evaluation across multiple model types.
- Findings: Closed-source frontier models outperform open-weight models, although open-weight models are often more cost-effective.The conclusion explicitly contrasts performance with cost-effectiveness.
- Limitations: The dataset is relatively modest in size and imbalanced across subtasks, so increasing samples per subtask would strengthen statistical reliability.The authors identify sample expansion as a way to improve the reliability of their analyses.
- Limitations: Because students primarily created questions to challenge two frontier models, the dataset may be biased toward those models’ specific weaknesses.This limitation concerns the construction process and the resulting scope of failure modes represented.
A Examples of problems
The benchmark includes visually grounded word extraction, directed shortest-path reasoning, and geometry tasks that combine image generation with problem solving. Each example is paired with explicit correctness criteria and task-specific failure modes.
- Word-search example: The word-search task requires extracting exactly ten colored words: WORD, SEARCH, JOURNEY, LOVE, RELAX, ESCAPE, FUN, PLAY, PUZZLE, and ENJOY.Answers omitting any listed word or adding an extra word are considered wrong.
- Graph example: The directed-graph task requires a minimum-weight directed path from A to B, with every step following an existing edge and the total weight being minimal.Common errors include reversing edge directions or inventing non-existent edges, yielding unreachable paths.
- Geometry example: The square-geometry task asks a model to generate an image and compute ABCD’s area from distances of 5 cm and 7 cm to line segment l.It combines abstract reasoning and object-centric processing, with geometric and graph reasoning plus attribute binding.
- Geometry example: Failure occurs when the generated diagram violates stated properties, such as placing B off AD or labeling the square ABDC, or when the model fails to compute the area.The former reflects attribute-binding failure, while the latter reflects geometric and graph-reasoning failure.
A.1 Task type analsysis … D Models Info List
The paper categorizes recurring blind-spot patterns, specifies automated grading prompts for text and image-generation tasks, validates grader alignment with human judgment, and documents the evaluated models. The taxonomy highlights difficulties in object-centric perception, multi-step abstract reasoning, and exact character-level manipulation.
- A.1 Task type analsysis: The taxonomy examines recurring patterns in intentionally challenging tasks designed to expose capabilities that remain difficult for modern AI models.Table 4 provides detailed descriptions and examples of the task taxonomy.
- A.1 Task type analsysis: Object-centric tasks reveal failures in attribute and pattern recognition because models may report training-distribution patterns instead of perceiving actual content.Attribute binding and generative counting can also require novel recombinations of familiar concepts that rarely occur in natural data.
- A.1 Task type analsysis: Abstract reasoning accuracy decreases as the number of inference steps increases, partly because locally greedy choices accumulate errors without global verification.The passage characterizes this as a reasoning gap in multi-step problems.
- A.1 Task type analsysis: Character-level manipulation fails under strict constraints such as exact character counts or repeated token sequences, especially when external tools are disallowed.The passage conjectures that subword tokenization limits exact character counting and length control.
- B Full Grader Prompts: Text-generation grading compares a submission with the question and ground-truth criterion, requiring a clearly stated final answer and assigning only C or I.Code, tool calls, search queries, reasoning, or derivation instructions without a clearly reported final answer are graded I; ambiguous or partial cases use best judgment.
- B Full Grader Prompts: Image-generation grading checks content accuracy, instruction following, quality, and relevance against the task requirements and grading criteria.The grader evaluates the generated image as the candidate answer and returns C when requirements are satisfied, otherwise I.
- C Grader Validation: Tables 5 and 6 report detailed metrics for verifying alignment between the automatic evaluation pipeline and human judgment.The passage identifies this as grader validation.
- D Models Info List: Table 7 collects the detailed specifications of all evaluated models.The passage presents this information as the models’ complete details.
E Full Models Performance
This section reports per-question model scores to reveal shared failures, using modality-specific heatmaps of mean@k with problems ordered by average accuracy across models.
- E Full Models Performance: Per-question scores identify shared model failures through heatmaps of mean@k for text-only, multi-to-text, and image-generation tasks.Problems are sorted by average accuracy across models; results appear in Figures 6, 7, and 8, respectively.
F Additional analysis
The paper presents additional analyses based on the proposed taxonomy in Table 8.
- Table 8 presents analyses based on the proposed taxonomy.
- The proposed taxonomy serves as the basis for the additional analyses.
- These taxonomy-based analyses appear in the paper’s additional analysis section.
G Broader Impact
blind-spots-bench is presented as a diagnostic benchmark for revealing persistent model failures that aggregate benchmark success may overlook. It may support targeted model comparisons and applicability evaluation, but its small, biased, and limited dataset prevents thorough reliability assessment.
- Diagnostic value: blind-spots-bench detects persistent language-understanding gaps by testing tasks humans solve easily but modern AI finds difficult.It is intended to clarify failure cases otherwise overlooked by aggregate success on other benchmarks.
- Safety and applicability: Minor errors in number or spatial reasoning, attribute binding, or instruction following could cause misinterpretations or unsafe actions.The benchmark may therefore inform applicability evaluations in education, assistive technologies, document parsing, and human-AI collaboration workflows.
- Model comparison: The benchmark may enable more insightful model comparisons by showing how different approaches behave within individual categories.Its usefulness is framed as category-level comparison rather than a complete reliability evaluation.
- Limitations: The benchmark should not be treated as a thorough reliability assessment because its dataset is small, biased toward particular problems, and limited in question formulation and answer distribution.These constraints mean the dataset will inevitably show signs of bias.
H License information
The benchmark dataset is distributed on HuggingFace under CC-BY-4.0 for research purposes, while evaluation code is provided separately under an Inspect AI MIT-licensed pipeline. Third-party packages retain their original licenses, and model outputs, evaluation logs, and generated images are excluded.
- Dataset license: The dataset includes benchmark prompts, reference solutions, task and sub-task annotations, failure-mode annotations, and associated metadata.It was suggested by graduate-level AI-course students and refined, filtered, and annotated by the authors.
- Dataset license: The dataset is available on HuggingFace under the CC-BY-4.0 license for research purposes.
- Evaluation software: The evaluation code and scripts are distributed separately, and the evaluation pipeline uses Inspect AI under the MIT License.
- Evaluation software: Third-party software packages retain their original license agreements, while model outputs, evaluation logs, and generated images are not included in the benchmark distribution.