Source-linked AI summary
SimpleVQA: Multimodal Factuality Evaluation for Multimodal Large Language Models
Xianfu Cheng, Wei Zhang, Shiwei Zhang, Jian Yang, Xiangyuan Guan, Xianjie Wu, Xiang Li, Ge Zhang, Jiaheng Liu, Yuying Mai, Yutao Zeng, Zhoufutu Wen, Ke Jin, Baorui Wang, Weixiao Zhou, Yunhong Lu, Tongliang Li, Wenhao Huang, Zhoujun Li
TL;DR
MLLMs require reliable evaluation of fact-based answers grounded in visual inputs, because factual inaccuracies and hallucinations remain a major concern. SimpleVQA provides a bilingual, multi-task benchmark with concise, stable answers and uses it to assess leading models and analyze error sources. The evaluation finds that current MLLMs generally perform poorly, with visual understanding and internalized knowledge both remaining important challenges.
Problem
Existing factuality benchmarks mainly evaluate text-only capabilities, leaving the image-grounded factuality of MLLMs insufficiently measured.
Method
SimpleVQA organizes 2,025 bilingual visual question-answer pairs across 9 tasks and 9 topics, then uses short-answer judging and atomic questions to analyze model errors.
Results
Current MLLMs generally perform poorly on SimpleVQA; the strongest reported overall F-scores are 56.3 for Chinese and 52.8 for English queries.
Takeaways & Limitations
Improving both visual understanding and internalized knowledge is identified as important for improving factual-answer accuracy, while hallucinations remain a substantial challenge.
Abstract
from arXiv · showhide
The increasing application of multi-modal large language models (MLLMs) across various sectors have spotlighted the essence of their output reliability and accuracy, particularly their ability to produce content grounded in factual information (e.g. common and domain-specific knowledge). In this work, we introduce SimpleVQA, the first comprehensive multi-modal benchmark to evaluate the factuality ability of MLLMs to answer natural language short questions. SimpleVQA is characterized by six key features: it covers multiple tasks and multiple scenarios, ensures high quality and challenging queries, maintains static and timeless reference answers, and is straightforward to evaluate. Our approach involves categorizing visual question-answering items into 9 different tasks around objective events or common knowledge and situating these within 9 topics. Rigorous quality control processes are implemented to guarantee high-quality, concise, and clear answers, facilitating evaluation with minimal variance via an LLM-as-a-judge scoring system. Using SimpleVQA, we perform a comprehensive assessment of leading 18 MLLMs and 8 text-only LLMs, delving into their image comprehension and text generation abilities by identifying and analyzing error cases.
1. Introduction
SimpleVQA addresses the need for reliable multimodal factuality evaluation by benchmarking concise visual question answering across diverse tasks and scenarios. The benchmark also supports systematic analysis of whether errors arise from visual understanding or internalized knowledge.
- SimpleVQA targets factual inaccuracies and hallucinations that limit the reliability and broader adoption of general-purpose AI systems.
- The benchmark contains 2,025 high-quality question-answer pairs spanning 9 topics and 9 application tasks.
- SimpleVQA supports bilingual Chinese-English evaluation, multiple VQA task forms, diversified scenarios, and concise short-answer assessment.
- Short answers enable LLM-based judging to determine correctness and compute an overall accuracy rate efficiently.
- The study evaluates 18 MLLMs and separates factual-question capability into visual understanding and internalized knowledge.
- SimpleVQA findings report insufficient factual accuracy, knowledge errors, overconfidence, and persistent challenges in image understanding and knowledge internalization.
2. SimpleVQA
SimpleVQA is a factual multimodal benchmark built from diverse, objective, challenging visual questions with concise, standardized answers. Its structured construction and validation process yields a dataset spanning multiple tasks and domains for efficient MLLM evaluation.
- Dataset scope: SimpleVQA organizes content across nine domains, including literature, education and sports, history and culture, contemporary society, technology, media, natural science, art, and life.The benchmark also includes separate Chinese history and culture coverage.
- Benchmark design: The benchmark targets factual knowledge boundaries rather than general vision-language understanding, using concise short-answer questions and standardized answers assessed by direct matching.Ideological and politically sensitive content is excluded, while questions and answers are designed for objective and efficient evaluation.
- Quality criteria: Questions, images, and answers are constrained by objectivity, verifiability, authenticity, temporal stability, and sufficient reasoning context.Questions require unique answers and evidence; images must be real, unaltered, text-free, and valid before August 2024.
- Data construction: The construction pipeline uses seed collection, GPT-4o-based data enhancement, LLM verification, difficulty screening, and atomic-fact extraction.Seeds come from existing VQA datasets and search engines, while generated candidates receive professional review.
- Quality control: From 8,360 initial pairs, multiple filtering and validation rounds retain 2,025 high-precision Q&A pairs, or 24% of the original dataset.The process removes insufficiently challenging samples and selects 200 difficult mathematical questions to preserve category balance.
- Dataset scope: SimpleVQA contains 2,025 samples across 9 tasks, 9 domains, and 244 image types.Its tasks include logic and science, object recognition, time and events, person and emotion, location and building, text processing, quantity and position, art and culture, and object attributes.
3. Experiments
The experiments evaluate 18 vision-language models with standardized settings and factuality metrics, finding substantial performance differences across tasks, languages, domains, and model families.
- The evaluation covers 18 models, including 8 closed-source and 10 open-source systems spanning diverse architectures and training paradigms.
- SimpleVQA reports Correct, Not Attempted, Incorrect, and Correct Given Attempted metrics to assess answer accuracy, reliability, and consistency.
- Gemini-2.0-flash and Doubaovision-pro-128k show strong task-level performance, especially on Time & Event and Person & Emotion.
- Most evaluated models perform poorly, revealing a clear gap between state-of-the-art and open-source systems except Qwen2.5-VL-72B-Instruct.
- Without image inputs, language models still achieve limited accuracy on image-dependent questions, with DeepSeek-R1 showing more prominent guessing ability.
4. Further Analysis
Further analysis separates visual understanding from internalized knowledge and uses atomic questions to investigate the sources of factual errors in MLLM responses.
- The analysis tests visual understanding and internalized knowledge as two distinct capabilities underlying factual question answering.
- The CFQ dataset contains 569 difficult examples designed to test whether models can recognize image content and use stored knowledge.
- LLMs struggle to activate internalized knowledge from atomic facts without images, while MLLMs show a mismatch between literacy and knowledge internalization.
- MLLM performance on atomic questions indicates potential for optimizing literacy through supervised fine-tuning.
5. Related Works
Related work spans multimodal benchmarks for visual-textual integration and factuality benchmarks for knowledge-grounded generation, while SimpleVQA uses a short-answer CFQ analysis.
- Existing vision-language benchmarks assess multimodal integration through tasks such as OCR, spatial awareness, retrieval, reasoning, and multi-task evaluation.
- Factuality benchmarks evaluate whether generated content follows commonsense, world knowledge, and domain-specific information, often against authoritative sources.
- In SimpleVQA’s CFQ experiments, Origin denotes original-answer Correct, Atomic denotes atomic-answer Correct, and Atomic-Given denotes original-answer Correct given atomic facts.
- MMLU evaluates multitask accuracy across 57 diverse tasks, illustrating the breadth of text-based factuality assessment.
6. Conclusion
SimpleVQA is introduced as a bilingual benchmark for factual visual question answering and is used to assess MLLMs while diagnosing visual and knowledge-related errors.
- SimpleVQA is a bilingual visual question-answering benchmark designed to evaluate fact-based quizzing capabilities in existing MLLMs.
- The benchmark supports Chinese and English, multiple tasks and scenes, challenging high-quality content, static references, and straightforward evaluation.
- The authors assess 18 MLLMs and 8 text-only LLMs and analyze errors through atomic questions targeting visual comprehension and visual-linguistic alignment.
- SimpleVQA is intended to support factuality assessment and the development of more trustworthy and reliable MLLMs.
A. Human Annotation cost.
Human annotators were compensated and supported with equipment, meals, and a comfortable working environment. They labeled about 2,025 questions, with total annotation costs of approximately $5,202.
- Annotators received the equivalent of $1 per question for their labeling work.
- The labeling setup included a comfortable working environment, free meals, souvenirs, computer equipment, and a GPT-4o interface.
- About 2,025 questions were labeled at a total cost of about $5,202 in US dollars.Annotators also checked derived tasks, including multilingual Q&A explanation and code completion.
B. Nine task categories SimpleVQA Smaples of SimpleVQA.
SimpleVQA samples are organized into nine task categories. Figure 5 presents examples of these categories.
- SimpleVQA contains nine task categories represented through benchmark samples.
- Figure 5 presents samples from the nine SimpleVQA task categories.
- The task-category samples provide examples of how SimpleVQA items are organized.
C. Results of Mainstream LLMs
The study reports CO, NA, IN, and CGA results for eight LLMs evaluated on SimpleVQA without image input.
- CO, NA, IN, and CGA results are reported for eight LLMs across SimpleVQA.
D. Results of Task Categories
SimpleVQA reports CO, 1-NA, IN, and CGA results for eight models across its nine task categories. The results are presented in Figures 6–9, with Table 6 additionally reporting no-image-input results for eight LLMs.
- CO, 1-NA, IN, and CGA results are reported for eight models across nine SimpleVQA task categories.
- Figure 6 presents CO results for eight different models across the nine task categories.
- Figure 7 presents 1-NA results, Figure 8 presents IN results, and Figure 9 presents CGA results across the nine task categories.
- Table 6 reports CO, NA, IN, and CGA results for eight LLMs on SimpleVQA without image input.
E. Results of Domain Categories
Figures 10–14 report five evaluation measures for eight models across nine domain categories.
- CO results compare eight models across nine domain categories.
- 1-NA results compare eight models across nine domain categories.
- IN results compare eight models across nine domain categories.
- CGA results compare eight models across nine domain categories.
- F-Score results compare eight models across nine domain categories.
F. Model Lists
The supplied material identifies close-sourced and open-sourced model tables, alongside annotation and quality-control procedures for constructing visual question-answering data.
- Model Lists: Tables 7 and 8 list the close-sourced and open-sourced models adopted in the experiments.
- Annotation Procedure: The annotation procedure rewrites paired judgment questions into visual question-answering prompts and extracts the answer associated with the “Yes” judgment.
- Quality Control: Valid questions must be image-grounded, grammatically coherent, nontrivial, and answerable from the image.
- Quality Control: A data entry is qualified only when both its rewritten question and answer are valid and relevant to the image.
- Benchmark Criteria: SimpleVQA questions require a single clear answer and time-invariant answers to support stable factual evaluation.
- Question Generation: Atomic-question generation identifies an entity from an original question and combines it with the entity’s class to formulate a logical image-question prompt.