Source-linked AI summary
WorldVQA: Measuring Atomic World Knowledge in Multimodal Large Language Models
Runjie Zhou, Youbo Shao, Haoyu Lu, Bowei Xing, Tongtong Bai, Yujie Chen, Jie Zhao, Lin Sui, Haotian Yao, Zijia Zhao, Hao Yang, Haoning Wu, Zaida Zhou, Jinguo Zhu, Zhiqi Huang, Yiping Bao, Yangyang Liu, Y. Charles, Xinyu Zhou
TL;DR
WorldVQA targets a gap in multimodal evaluation: existing benchmarks often mix visual knowledge retrieval with reasoning or OCR-related dependencies. It introduces an atomic entity-naming benchmark spanning head and long-tail categories, and current models remain below 50% overall performance, revealing substantial visual-knowledge headroom.
Problem
Existing multimodal evaluations often conflate visual recognition with reasoning or secondary dependencies, making atomic visual knowledge difficult to measure.
Method
WorldVQA evaluates direct, unassisted naming of visual entities across nine semantic categories while excluding OCR, arithmetic, and multi-hop reasoning.
Results
No evaluated model exceeds 50% overall performance, while Nature and Culture are notable weaknesses and Brands and Sports are stronger categories.
Takeaways & Limitations
WorldVQA provides a precise benchmark for visual knowledge grounding, factual reliability, and hallucination assessment in MLLMs.
Takeaways & Limitations
It remains open whether performance on atomic entity naming correlates strongly with complex downstream multimodal tasks.
Abstract
from arXiv · showhide
We introduce WorldVQA, a benchmark designed to evaluate the atomic visual world knowledge of Multimodal Large Language Models (MLLMs). Unlike current evaluations, which often conflate visual knowledge retrieval with reasoning, WorldVQA decouples these capabilities to strictly measure "what the model memorizes." The benchmark assesses the atomic capability of grounding and naming visual entities across a stratified taxonomy, spanning from common head-class objects to long-tail rarities. We expect WorldVQA to serve as a rigorous test for visual factuality, thereby establishing a standard for assessing the encyclopedic breadth and hallucination rates of current and next-generation frontier models.
1 Introduction
WorldVQA addresses the difficulty of measuring visual knowledge separately from reasoning by testing direct entity naming across a broad taxonomy. It is designed to expose current models’ limited and uneven visual grounding, with no evaluated model exceeding 50% accuracy.
- Motivation: Existing VQA benchmarks often conflate visual recognition with multi-step reasoning, language knowledge, or OCR dependencies.This makes it difficult to identify whether an error reflects missing visual knowledge or another capability.
- Benchmark design: WorldVQA evaluates the direct association between an unambiguous visual stimulus and its specific proper noun or taxonomic name.The benchmark excludes OCR, arithmetic, and multi-hop retrieval to isolate visual grounding.
- Benchmark design: The benchmark spans nine semantic categories and balances high-frequency head entities with rare long-tail instances to profile visual-memory breadth.Categories include Nature, Geography, Culture, Objects, Transportation, Entertainment, Brands, Sports, and People.
- Findings: No evaluated model surpasses the 50% accuracy threshold, indicating substantial headroom for grounding atomic visual knowledge.Figure 1 reports Gemini-3-pro at 47.4% and Kimi K2.5 at 46.3%.
- Contribution: WorldVQA provides a precise metric for visual hallucination and knowledge grounding and is released as a standard for evaluating factual reliability.Its stated purpose is to assess next-generation MLLMs’ factual reliability.
2 Data Collection and Verification
WorldVQA is constructed through taxonomically controlled collection, balancing, deduplication, model-based difficulty stratification, and dual human–model verification. The resulting benchmark contains 3,500 pairs with balanced linguistic and categorical coverage and a 1:1.78 Chinese-to-English ratio.
- Collection: Ten expert annotators curate atomic entities and VQA pairs according to taxonomic, granularity, and visual-reliability standards.The pipeline begins with expert selection of entities, images, and questions from internal lexicons and trusted web sources.
- Curation criteria: WorldVQA enforces single-hop entity identification while excluding OCR, arithmetic, and external logical deduction.Answers must match the target’s required taxonomic specificity rather than a generic hypernym.
- Curation criteria: The dataset balances global representation by capping entities unique to the Chinese context at 50% per category and expanding underrepresented categories.LLM-in-the-loop association search is used for global entity expansion.
- Deduplication: A 0.95 ISC cosine-similarity threshold removes near-duplicate images found against LAION and Common Crawl.This step is intended to mitigate redundancy and leakage from common pre-training corpora.
- Difficulty stratification: Candidate samples are stratified by ensemble performance into Easy, Medium, and Hard tiers, with Easy samples downsampled and Hard samples human-reviewed.The tiers correspond to >3, 1–2, and 0 models correct, respectively.
- Benchmark statistics: The final benchmark contains 3,500 pairs with balanced linguistic and categorical spread and a 1:1.78 Chinese-to-English ratio.These statistics are presented as the benchmark’s high-level composition.
3 Experiments
WorldVQA evaluation shows that frontier MLLMs remain below encyclopedic visual proficiency, with performance varying substantially across categories and entity rarity. Calibration analysis further indicates widespread overconfidence and weak alignment between stated confidence and actual accuracy.
- Main Results: 47.5% F-score is the highest reported overall result, achieved by Gemini-3-pro, while Kimi K2.5 reaches 46.8% as the top open-source model.No model surpasses the 50% threshold on WorldVQA.
- Main Results: Brands and Sports are relative strengths, whereas Nature and Culture are significant weaknesses in category-wise performance.Gemini-3-pro achieves an F-score of 59.4 in Sports; models often produce generic hypernyms instead of specific entities in weaker domains.
- Main Results: 29.3% CGA versus 26.7% F-score for GPT-5.1 indicates conservative answering, while smaller models more often hallucinate obscure entity names.The CGA–F-score mismatch is used as a probe of model honesty and knowledge-boundary awareness.
- Validation of Difficulty Stratification: Trivial and Easy entities cluster near the highest-frequency percentile, while Medium and Hard entities shift toward rarer MetaCLIP vocabulary ranks.The fitted distributions show a positive correlation between real-world rarity and benchmark difficulty, supporting WorldVQA’s long-tail stratification.
- Calibration Analysis: 37.9% ECE and 0.550 Slope make Kimi K2.5 the best-calibrated model, although all models exhibit severe overconfidence.ECE measures confidence–accuracy alignment, while Slope assesses the correlation between accuracy and confidence.
- Calibration Analysis: Gemini-3-pro assigns at least 95% confidence in over 85% of cases, while GPT-5.1 distinguishes low confidence more often despite a slightly higher ECE.Most models concentrate stated confidence in the 90–100% range.
4 Related Work and Discussion
WorldVQA is positioned as a benchmark for isolating visual factuality from reasoning and examining multimodal reliability. Its atomic design also leaves open how entity naming relates to complex downstream performance and RL-induced calibration.
- Related benchmarks: WorldVQA targets atomic visual factuality within a broader MLLM evaluation landscape that includes holistic competence benchmarks.Existing suites often treat world knowledge as an implicit prerequisite rather than isolating it directly.
- Reliability: Prior work motivates examining calibration and hallucination because latent self-knowledge may be distorted by post-training alignment.The cited literature provides methodological grounding for reliability analysis in language and multimodal settings.
- Discussion and limitations: Whether atomic entity naming correlates strongly with complex downstream multimodal performance remains an open research question.The paper also has not fully quantified how different RL strategies affect atomic visual calibration.
A Prompts
The prompts operationalize answer correctness and visual support through explicit classification rules. They distinguish correct, incorrect, and unattempted answers while requiring visual judgments based on clarity, exclusivity, and completeness.
- Visual fact-checking: Visual fact-checking requires judging whether an image alone provides unique, conclusive, and exclusive evidence for an answer.The procedure checks clarity, uniqueness or exclusivity, and completeness before assigning Determinable, Inconclusive, or Incorrect.
- Visual fact-checking: The prompt frames the evaluator as an expert judge assessing answer correctness against the available visual evidence.This role complements the separate answer-correctness and visual-support criteria.
- Answer evaluation: Answer correctness is classified as Correct, Incorrect, or Unattempted using semantic consistency and answer granularity.Correct answers must preserve the ground truth’s core information without contradiction and meet or exceed its specificity.
- Answer evaluation: The evaluation format requires a brief reasoning statement followed by a final classification label.The prescribed output uses two lines: an evaluation explanation and one of the three labels.
B WorldVQA Showcases
The showcase presents a bird-identification question whose answer is the specific entity “Chestnut Shortwing.”
- Bird example: The showcase asks which bird appears in the picture and gives “Chestnut Shortwing” as the answer.The example illustrates entity naming at a specific taxonomic level.
B.2 Locations & Architecture
The showcase questions span flowers, landmarks, performances, bags, electronics, characters, vehicle accessories, and screen media. One location example identifies the natural landmark as “Cape of Good Hope.”
- Locations and architecture: The showcase includes questions about flowers, natural landmarks, dance performances, bag styles, and electronic products.These prompts cover multiple visual entity and attribute types.
- Locations and architecture: “Cape of Good Hope” is provided as the answer to the natural-landmark identification question.The example demonstrates a proper-name answer for a landmark.
- Locations and architecture: The examples also ask for character identities, vehicle attachments or accessories, and the film or television series depicted.These prompts extend the showcase to entertainment and vehicle-related identification.
B.7 Brands, Logos & Graphic Design
The section tests visual recognition of advertisement media and records direct-mail advertisement as the answer.
- The advertisement medium shown is direct mail.
B.8 Sports, Gear & Venues
The section contains visual questions about identifying a track-and-field or gymnastics event and naming a trademark or logo.
- One prompt asks for the most specific identification of a track-and-field or gymnastics event.
- Another prompt asks for the name of a trademark or logo shown in an image.