PaperlayerView exact revision on arXiv

Source-linked AI summary

WorldBench: A Challenging and Visually Diverse Multimodal Reasoning Benchmark

Yida Yin, Harish Krishnakumar, Chung Peng Lee, Boya Zeng, Wenhao Chai, Shengbang Tong, Wenhu Chen, Hu Xu, Xingyu Fu, Gabriel Sarch, Aleksandra Korolova, Zhuang Liu

arXiv:2606.06538v1cs.CV

TL;DR

Task-centric multimodal benchmarks can overlook the visual diversity needed to evaluate varied image-grounded abilities; WorldBench instead builds around visually diverse images and challenging questions. Evaluations find greater visual diversity than existing diverse benchmarks and difficulty for current MLLMs, with Gemini-3.1-Pro achieving 64.0% average accuracy.

  • Problem

    Task-centric benchmarks overlook image diversity, which shapes the questions asked and the model abilities evaluated.

    Source: §1 · p. 1

  • Method

    WorldBench uses a taxonomy of thousands of concepts across seven domains to curate diverse images and iteratively refine questions until a frontier MLLM fails.

    Source: §1 · p. 2

  • Results

    WorldBench exhibits greater visual diversity than existing diverse benchmarks and challenges current MLLMs; Gemini-3.1-Pro achieves the highest accuracy of 64.0% averaged across all domains.

    Source: §1 · p. 2, §6 · p. 12

  • Takeaways & Limitations

    The findings motivate building multimodal datasets that prioritize visual diversity.

    Source: §6 · p. 12

    Using a pre-trained vision encoder as a general visual-diversity metric is limited, with participation-ratio rankings varying across encoders.

    Source: §3.1 · p. 8

Abstract

from arXiv · show

In real-world applications, models are expected to perform reliably across diverse settings. Yet, many existing multimodal benchmarks expand task types without capturing the visual diversity needed to handle open-ended visual inputs. We present WorldBench, a challenging and visually diverse reasoning benchmark to evaluate Multimodal Large Language Models (MLLMs). We build a taxonomy of thousands of visual concepts across multiple domains (e.g., living things). Guided by this taxonomy, we curate a broad collection of images from search engines and existing datasets to comprehensively represent the visual world. Through structured trial-and-error, we manually design challenging questions that frontier MLLMs fail to answer. On quantitative and human evaluations, WorldBench achieves higher visual diversity than any existing diverse benchmark. Evaluating 15 MLLMs on WorldBench reveals weaknesses in visual understanding: even the strongest model reaches only 64.0% accuracy, while some models perform marginally above chance-level. We hope our work highlights the importance of visual diversity in building multimodal benchmarks.

1 Introduction

WorldBench builds a visually diverse, challenging MLLM benchmark around broad coverage of the visual world rather than task categories. It ranks highly in diversity evaluations, yet current models remain limited in accuracy and visual grounding.

1 Introduction

  • Task-centric benchmarks overlook image diversity, so WorldBench instead pairs diverse images with questions intuitive to people but challenging for models.The paper argues image variety shapes which questions can be asked and, in turn, which model abilities can be evaluated.Source: §1 · p. 1
  • Its taxonomy spans thousands of fine-grained concepts across 7 visual domains and guides curation of 2,000 questions from web searches and existing datasets.The semi-automated taxonomy uses light human effort; curated images favor rich contextual scenes, and questions are iteratively refined until at least one frontier MLLM fails.Source: §1 · p. 2
  • WorldBench ranks among the top two benchmarks across vision encoders and earns the highest overall human-rated diversity score.Quantitative diversity uses effective rank and participation ratio of image-embedding feature covariance; human judgments aggregate pairwise votes with the Bradley–Terry model.Source: §1 · p. 2, Figure 1 · p. 2
  • Gemini-3.1-Pro reaches 64.0% average accuracy, while open-source Qwen3.5-VL-27B reaches 56.6%; no model exceeds 75% in any domain.The evaluation covers 15 MLLMs.Source: §1 · p. 2
  • Models struggle with fine-grained perception and make ungrounded inferences; reasoning variants outperform non-reasoning ones, but more reasoning tokens do not always help.These patterns are observed across all domains.Source: §1 · p. 2

2 WorldBench

WorldBench addresses the lack of visual diversity in multimodal benchmarks with a broad image collection and questions designed to challenge frontier MLLMs while remaining intuitive for humans.

2.1 Building a Comprehensive Taxonomy

  • WorldBench’s taxonomy covers 2,000 visual concepts across seven domains, extending beyond object-centric categories to web-native content through LLM-assisted expansion and human refinement.For Robotics, the authors derive concepts from robot-centric images in AgentVQA.Source: §2.1 · pp. 3–4, Figure 3 · p. 4

2.2 Curating Images from Taxonomy

  • WorldBench’s 2,000 images offer greater visual diversity and a more balanced concept distribution than MMMU (Yue et al., 2024), MEGA-Bench (Chen et al., 2025), and MMBench.Images are selected to match taxonomy concepts, prioritizing non-iconic views with richer context; Robotics images are selected from AgentVQA and human-verified.Source: §2.2 · pp. 4–5, Figure 4 · p. 5

2.3 Proposing Challenging Questions

  • For each image, the authors iteratively revise a four-option question until at least one evaluated frontier model answers incorrectly.Questions need not directly test the image’s retrieval concept, and annotators may provide concise explanations for challenging items.Source: §2.3 · pp. 5–6
  • Manual review and Claude Code-assisted auditing target ambiguous wording, multiple valid answers, and inaccurate image descriptions to ensure each question has one valid answer.Volunteers audit 100 sampled questions; proposed revisions are adopted only when reviewers agree they meaningfully improve quality.Source: §2.3 · p. 6
  • Unlike ZeroBench (Roberts et al., 2025), which chains independent subquestions, WorldBench aims for natural, real-world questions that are intuitive to humans but often answered incorrectly by frontier models.The paper illustrates this contrast with six representative question samples.Source: §2.3 · pp. 6–7

3 Measuring Visual Diversity of Images

WorldBench’s visual diversity is assessed using pretrained-encoder embeddings and pairwise human judgments. It often ranks among the most diverse benchmarks quantitatively and receives the highest human rating, though quantitative rankings vary by encoder.

3.1 Quantitative Evaluation

  • The quantitative method samples 1,000 images per benchmark and measures covariance-based effective rank, with more uniform embedding variance indicating greater diversity.The evaluation uses multiple pretrained encoders: SigLIP 2, Perception Encoder, and DINO v3, trained on distinct data distributions.Source: §3.1 · pp. 7–8
  • Across three vision encoders, WorldBench often ranks first or second on effective rank and participation ratio.The authors interpret this as evidence that WorldBench maintains greater visual diversity across different visual representations.Source: §3.1 · p. 8
  • Quantitative diversity rankings vary considerably by encoder: MMBench leads participation ratio with SigLIP 2 but ranks fourth with Perception Encoder.The authors note that this variation suggests encoders capture distinctive visual representations.Source: §3.1 · p. 8

3.2 Human Evaluation

  • The human study compares panels of 100 randomly sampled images through pairwise votes, collecting 360 comparisons from 12 volunteers.Benchmark scores are estimated with a Bradley–Terry model; bootstrap confidence intervals use 10,000 resampling rounds.Source: §3.2 · pp. 8–9
  • WorldBench receives the highest human-rated visual-diversity score among the evaluated benchmarks.Figure 7 reports Bradley–Terry scores and 95% confidence intervals from pairwise human comparisons.Source: Figure 7 · p. 9, §3.2 · p. 9
  • Users judged diversity using semantic and category breadth, balance of charts or diagrams versus photos, visual variety, and the absence of duplicates or near-duplicates.The authors caution that some criteria, such as semantic content and category breadth, depend more on individual judgment than identifying duplicates.Source: §3.2 · p. 9

4 Model Performance

WorldBench remains challenging for evaluated MLLMs, and its analysis finds that reasoning-budget effects vary by domain while its scores correlate least with other diverse benchmarks.

4.1 Settings

  • The evaluation covers 15 MLLMs using a unified multiple-choice prompt and fixed answer-parsing rules, with default model-specific generation settings.Open-source models are evaluated offline with vLLM except Kimi-K2.5, which is evaluated through its API.Source: §4.1 · p. 10

4.2 Results

  • Gemini-3.1-Pro leads proprietary models at 64.0% average accuracy, while Qwen3.5-VL-27B leads open-source models at 56.6%, and no model exceeds 75% in any domain.Qwen3.5-VL-27B also outperforms Claude-Opus-4.7 and Grok-4.2, which are often considered more capable.Source: §4.2 · p. 10

4.3 Analysis

  • With GPT-5.4, low reasoning improves accuracy across all domains; higher budgets keep lifting Digital World, Objects, Scenes, and Agents but saturate or regress in four other domains, sometimes causing reasoning loops.The affected domains are Documents, Charts, & Tables; Academics; and Living Things.Source: §4.3 · p. 11, Figure 8 · p. 11
  • WorldBench has the lowest average Pearson correlation with the other selected benchmarks, at 0.94, suggesting its visual-diversity emphasis assesses a different dimension.The comparison uses model accuracies on WorldBench, MMMU, MMStar, and MMT-Bench.Source: §4.3 · p. 11, Figure 9 · p. 11

5 Related Work

Prior MLLMs have advanced rapidly, but many benchmarks still use visually limited image sets or focus on particular tasks. WorldBench instead centers evaluation on visual diversity.

Multimodal large language models

  • Despite rapid proprietary and open-source MLLM progress, existing benchmarks often use image sets with limited visual diversity.The cited examples of proprietary and open-source models include OpenAI, 2026; Google, 2026; Anthropic, 2026; xAI, 2026; Qwen, 2026; GLM, 2025a; and Wang et al., 2025.Source: §5 · p. 11

Multimodal benchmarks

  • Unlike task-specific and broad task-coverage benchmarks, WorldBench departs from task-centric design by centering evaluation on visual diversity.Prior benchmarks target perception, OCR, hallucination, GUI navigation, and coding, while others broaden coverage across tasks.Source: §5 · p. 11

6 Conclusion · A Question Annotation Interface

WorldBench combines a broad taxonomy-driven image collection with questions designed to expose frontier MLLM failures; evaluations find it more visually diverse than existing diverse benchmarks and challenging for current models. Its annotation interface supports question generation, model testing, clarification, and wording refinement.

6 Conclusion

  • The benchmark builds a taxonomy of thousands of visual concepts and uses it to curate a diverse image set.Source: §6 · p. 11
  • WorldBench is more visually diverse than existing diverse benchmarks and remains challenging for current models, according to quantitative and human evaluations.The authors hope this motivates datasets that prioritize visual diversity.Source: §6 · p. 12
  • Questions are designed through structured trial and error to target frontier MLLM failure modes.Source: §6 · p. 12

A Question Annotation Interface

B Additional Results

Vision encoders rank pre-training datasets differently, with higher diversity scores tending to favor datasets resembling their training distributions. The results also highlight limits of single-representation diversity measures and the benchmark’s web-derived data.

B Additional Results

  • The evaluation samples 1,000 images from each of six pre-training datasets and measures diversity using effective rank and participation ratio across pre-trained vision encoders.The datasets are YFCC, CC, DataComp, WIT, LAION, and ImageNet.Source: B Additional Results · p. 16, Table 3 · p. 16
  • SigLIP2 ranks DataComp most diverse, while DINO v3 ranks ImageNet highest; scores tend to favor datasets resembling each encoder’s pre-training distribution.The authors attribute this pattern to differences in encoder training data: SigLIP2 uses web-sourced WebLI, while ImageNet makes up 1/10 of DINO v3’s pre-training data.Source: Table 3 · p. 16, B Additional Results · p. 17
  • Visual diversity is difficult to capture with a single metric or representation, and human evaluation is complementary but needs broader, more demographically diverse judgments.Because WorldBench draws on web search results and existing datasets, it may inherit their biases and is not sufficient evidence of safe or fair real-world deployment.Source: B Additional Results · p. 17

C User Study Interface for Human Evaluation

The human evaluation compared visual diversity by showing users image samples from two benchmarks side by side and asking them to choose the more diverse panel.

C User Study Interface for Human Evaluation

D Implementation Details · D.1 Benchmarks Used in Diversity Evaluation

The appendix specifies how images and benchmark splits were selected for the diversity evaluation. It also describes image licensing and content-safety review during WorldBench curation.

D.1 Benchmarks Used in Diversity Evaluation

  • To avoid disadvantaging WorldBench, the evaluation limits repeated or similar images that would reduce other benchmarks’ visual diversity.Some evaluated benchmarks reuse identical images across questions or include multiple similar images within one question.Source: §D.1 · p. 17
  • For each benchmark, the evaluation randomly samples one image per question and removes exact duplicates from the resulting set.Source: §D.1 · p. 18
  • For MEGA-Bench, sampling excludes in-context examples and uses only images on which models are evaluated.Source: §D.1 · p. 18
  • The evaluation uses validation VQAv2 (Goyal et al., 2017), full MME, and full MMStar.Source: §D.1 · p. 18
  • It also uses test MMBench (Yu et al., 2024a), test MMMU (Yue et al., 2024), and full SEED-Bench-2 (Li et al., 2023a).Source: §D.1 · p. 18
  • The remaining benchmarks are test MMT-Bench (Ying et al., 2024) and full MEGA-Bench (Chen et al., 2025).Source: §D.1 · p. 18
  • WorldBench curation filters web-search results for Creative Commons-licensed images and manually reviews images and questions to remove unsafe, private, or otherwise inappropriate content before release.Source: §D.1 · p. 18

D.2 Model Performance · E Model Responses

The sections describe how WorldBench responses were generated and parsed, and provide representative outputs from all 15 evaluated models. The evaluation uses model-specific default settings, a prompt that requests an explanation before the answer, and a parsing procedure with over 99% average success.

D.2 Model Performance

  • Proprietary models were evaluated with default sampling settings, except that exposed reasoning-effort settings were reported in the model name.The evaluated models were GPT-5.4-Thinking, Gemini-3.1-Pro, Gemini-3-Flash, Claude-Opus-4.7, Grok-4.2, and Qwen3.5-VL-Plus-Thinking/Instruct.Source: §D.2 · p. 18
  • Local open-source evaluations used vLLM, each model’s default or recommended generation parameters, and approximately 4–8 A100 GPUs for roughly one hour; Kimi-K2.5 was evaluated through its API.The local inference parameters are reported in Table 4, which excludes Kimi-K2.5.Source: §D.2 · p. 18, Table 4 · p. 18
  • Requesting a brief explanation before the final answer slightly improved accuracy over asking for the answer first, when models sometimes changed their answer during the explanation.Figure 12 illustrates the prompt format.Source: §D.2 · p. 18, Figure 12 · p. 19
  • Over 99% average parsing success across WorldBench questions was achieved with sequential rules that extract an explicit answer, a leading option letter, or a standalone option letter.Responses that matched none of the rules were marked incorrect, so the authors did not use an LLM-as-a-judge.Source: §D.2 · p. 18

E Model Responses

  • Figures 13–27 provide one representative response for each of the 15 models evaluated in Section 4.The examples include exposed reasoning traces when available; some thinking models do not return private traces through their APIs.Source: E Model Responses · p. 18
  • Some thinking-model examples omit private reasoning traces because those traces are not returned through the API.The figures include exposed traces when available.Source: E Model Responses · p. 18

Related papers

Loading 2606.06538v1…