Source-linked AI summary
SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension
Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, Ying Shan
TL;DR
MLLMs lacked a large, objective benchmark for generative comprehension across image and video understanding. SEED-Bench introduces 19K human-annotated multiple-choice questions across 12 dimensions, generated and verified through an automated and manual pipeline, and evaluates 18 models to expose limitations in existing multimodal systems.
Problem
Existing MLLM evaluations rely on limited qualitative examples, subjective human or GPT assessment, or small benchmarks that do not comprehensively cover generative comprehension.
Method
SEED-Bench constructs dimension-specific multiple-choice questions from visual information or video annotations, then applies automatic filtering and human verification.
Results
19K human-annotated questions span 12 dimensions, and 18 models are evaluated across spatial and temporal understanding.
Takeaways & Limitations
SEED-Bench provides an objective benchmark intended to support comparison of MLLM capabilities and motivate future research.
Takeaways & Limitations
Existing VideoLLMs do not show a significant advantage in temporal understanding and remain limited in fine-grained action recognition, temporal relationships, and temporal reasoning.
Abstract
from arXiv · showhide
Based on powerful Large Language Models (LLMs), recent generative Multimodal Large Language Models (MLLMs) have gained prominence as a pivotal research area, exhibiting remarkable capability for both comprehension and generation. In this work, we address the evaluation of generative comprehension in MLLMs as a preliminary step towards a comprehensive assessment of generative models, by introducing a benchmark named SEED-Bench. SEED-Bench consists of 19K multiple choice questions with accurate human annotations (x 6 larger than existing benchmarks), which spans 12 evaluation dimensions including the comprehension of both the image and video modality. We develop an advanced pipeline for generating multiple-choice questions that target specific evaluation dimensions, integrating both automatic filtering and manual verification processes. Multiple-choice questions with groundtruth options derived from human annotation enables an objective and efficient assessment of model performance, eliminating the need for human or GPT intervention during evaluation. We further evaluate the performance of 18 models across all 12 dimensions, covering both the spatial and temporal understanding. By revealing the limitations of existing MLLMs through evaluation results, we aim for SEED-Bench to provide insights for motivating future research. We will launch and consistently maintain a leaderboard to provide a platform for the community to assess and investigate model capability.
1 Introduction
Existing MLLM evaluations are limited by qualitative examples, exact-match open-form outputs, subjective human or GPT assessment, and small objective benchmarks. SEED-Bench addresses these gaps with a large, human-annotated, dimension-specific benchmark and evaluates models across spatial and temporal understanding.
- Current MLLM evaluations often rely on few qualitative examples or benchmarks not designed for open-form generative outputs.
- Human or GPT assessment can reduce evaluation efficiency and objectivity, while existing objective benchmarks contain fewer than 3K samples.
- SEED-Bench contains 19K human-annotated multiple-choice questions spanning 12 dimensions across image and video modalities.
- Its question-generation pipeline targets specific dimensions and combines foundation-model visual extraction, ChatGPT/GPT-4 generation, automated filtering, and human verification.
- SEED-Bench is designed to scale across domains and expand with additional evaluation dimensions.
- The benchmark evaluates 18 models across all 12 dimensions, using candidate-option log-likelihoods rather than requiring models to output option letters.
2 Related Work
Related MLLM work has expanded multimodal and video understanding, while existing benchmarks vary in coverage and evaluation procedures. SEED-Bench contributes a broader quantitative evaluation centered on generative comprehension.
- Generative MLLMs build on LLM capabilities to improve multimodal comprehension and generation, with some work extending to video inputs.
- SEED-Bench provides comprehensive quantitative evaluation to assess and compare MLLM performance in generative comprehension.
- Prior MLLM benchmarks combine existing computer-vision tasks or datasets, but their coverage can remain limited across visual-understanding dimensions.
3 SEED-Bench
SEED-Bench organizes image and video comprehension into 12 dimensions and constructs human-verified multiple-choice questions through a staged generation and verification pipeline. Its data sources and prompts are tailored to spatial, temporal, and reasoning requirements.
- Evaluation Dimensions: SEED-Bench covers 12 evaluation dimensions, comprising nine spatial dimensions and three temporal dimensions.
- Evaluation Dimensions: Spatial dimensions assess scene understanding, instance identity, attributes, location, counting, spatial relations, interaction, and visual reasoning.
- Evaluation Dimensions: Temporal dimensions evaluate action recognition, action prediction, and procedure understanding from videos.
- Data Source: The benchmark uses filtered CC3M images for spatial questions and SSV2, Epic-Kitchen 100, and Breakfast videos for temporal questions.
- Multiple-Choice Questions: Foundation models extract captions, instance descriptions, and textual elements before ChatGPT/GPT-4 generates dimension-specific questions with four options and one correct answer.
- Multiple-Choice Questions: Automatic filtering removes questions answerable without images, while human annotators discard unanswerable or ambiguously labeled items and categorize the remainder.
4 Evaluation Results
SEED-Bench evaluates 18 models across 12 spatial and temporal dimensions, revealing broad performance limits alongside distinct strengths across models and tasks.
- Evaluation setup: 18 models are evaluated across all 12 SEED-Bench dimensions, covering spatial and temporal understanding.The evaluation includes 3 LLMs, 12 ImageLLMs, and 3 VideoLLMs.
- Model-specific results: InstructBLIP achieves the best averaged spatial performance across nine dimensions and surpasses VideoLLMs on averaged temporal understanding.The model also ranks first on 8 of 12 individual evaluation dimensions.
- Model-specific results: Model strengths vary by dimension: LLaVa excels at text recognition, MiniGPT-4 and mPLUG-Owl at visual reasoning, and VPGTrans at action recognition and procedure understanding.The BLIP series is competitive across multiple dimensions but performs poorly in visual reasoning and action recognition.
- Overall performance: Most MLLMs cannot reach 50% accuracy on both average performance and more than three individual dimensions.The results indicate limited performance across the benchmark, despite higher accuracy in some specific dimensions.
- Global image comprehension: All MLLMs outperform LLMs on scene understanding and visual reasoning, where most MLLMs exceed 40% accuracy.These results indicate stronger global image comprehension than fine-grained instance-level comprehension.
- Temporal understanding: VideoLLMs do not achieve a significant advantage in temporal understanding, with two performing worse than most ImageLLMs on several temporal tasks.The affected dimensions are action recognition, action prediction, and procedure understanding.
5 Conclusion
SEED-Bench provides a large-scale, comprehensive, and objective evaluation of MLLMs’ generative comprehension across spatial and temporal understanding.
- SEED-Bench contains 19K multiple-choice questions with accurate human annotations across 12 spatial and temporal evaluation dimensions.The benchmark covers both image and video modalities.
- An advanced question-generation pipeline targets specific evaluation dimensions and combines automatic filtering with manual verification.
- Evaluation of 18 models compares performance across the benchmark’s dimensions to provide insights for future research.
- The authors plan to maintain a leaderboard and broaden SEED-Bench with additional evaluation dimensions and data.