Source-linked AI summary
WorldSense: Evaluating Real-world Omnimodal Understanding for Multimodal LLMs
Jack Hong, Shilin Yan, Jiayin Cai, Xiaolong Jiang, Yao Hu, Weidi Xie
TL;DR
Existing multimodal benchmarks inadequately evaluate tightly coupled audio-visual reasoning in diverse real-world contexts. WorldSense addresses this gap with a broad, carefully annotated benchmark and finds substantial limitations in current MLLMs, with Gemini 2.5 Pro reaching 65.1% accuracy.
Problem
Existing evaluations often focus on visual-language information and provide limited coverage of integrated audio-visual real-world understanding.
Method
WorldSense evaluates MLLMs using synchronized audio-visual videos, diverse cognitive tasks, and questions curated through expert annotation and multimodal quality control.
Results
65.1% accuracy is the highest reported result, achieved by Gemini 2.5 Pro, while open-source video-audio models achieve 25% accuracy.
Takeaways & Limitations
WorldSense exposes substantial limitations in processing and coherently integrating omnimodal information, while identifying raw audio and visual cues as important performance factors.
Takeaways & Limitations
Failure analysis finds that most errors arise from audio-understanding deficiencies and reasoning failures, with models often processing modalities independently.
Abstract
from arXiv · showhide
We introduce WorldSense, the first benchmark to assess the multi-modal video understanding, that simultaneously encompasses visual, audio, and text inputs. In contrast to existing benchmarks, our WorldSense has several features: (i)collaboration of omni-modality, we design the evaluation tasks to feature a strong coupling of audio and video, requiring models to effectively utilize the synergistic perception of omni-modality; (ii)diversity of videos and tasks, WorldSense encompasses a diverse collection of 1,662 audio-visual synchronised videos, systematically categorized into 8 primary domains and 67 fine-grained subcategories to cover the broad scenarios, and 3,172 multi-choice QA pairs across 26 distinct tasks to enable the comprehensive evaluation; (iii)high-quality annotations, all the QA pairs are manually labeled by 80 expert annotators with multiple rounds of correction to ensure quality. Based on our WorldSense, we extensively evaluate various state-of-the-art models. The experimental results indicate that existing models face significant challenges in understanding real-world scenarios (65.1% best accuracy). By analyzing the limitations of current models, we aim to provide valuable insight to guide development of real-world understanding. We hope our WorldSense can provide a platform for evaluating the ability in constructing and understanding coherent contexts from omni-modality.
1 Introduction
WorldSense addresses the limited evaluation of audio-visual understanding by requiring tightly coupled modalities for real-world video reasoning. It combines broad benchmark coverage and expert annotations, while evaluations reveal substantial limitations in current MLLMs.
- Existing multimodal evaluations primarily emphasize visual-language inputs, while audio-visual benchmarks often remain narrow, caption-focused, or simplistic.
- WorldSense requires both audio and visual modalities for each question, because removing either modality can prevent a correct response.
- 1,662 synchronized audio-visual videos span 8 domains and 67 subcategories, with 3,172 multiple-choice questions across 26 cognitive tasks.
- 80 expert annotators curate the question-answer pairs through multiple validation rounds involving human review and automated MLLM verification.
- 65.1% accuracy is achieved by the best proprietary model, while open-source video-audio models reach only 25%, comparable to random guessing.
- Ablations identify visual inputs as essential, with raw audio providing additional gains over text transcriptions.
2 Related Work
Prior multimodal research has expanded beyond vision-language modeling, but benchmark coverage still inadequately represents diverse, synchronized audio-visual real-world understanding.
- Multimodal language models have progressed across visual, textual, auditory, and temporal processing, including systems that jointly handle text, vision, and audio.
- Existing benchmarks largely overlook audio, while audio-visual alternatives often use static images, domain-specific tasks, monotonous questions, or caption-only evaluation.
- WorldSense responds with synchronized audio-visual videos and a data collection and QA annotation pipeline.
3 WorldSense
WorldSense is constructed from synchronized, diverse real-world videos and evaluated through multimodal questions spanning recognition, understanding, and reasoning. Its curation and quality-control procedures emphasize audio-visual necessity and benchmark difficulty.
- 3.1 Design Principle: WorldSense organizes audio-visual evaluation around comprehensive domain coverage, diverse acoustic modalities, and multilevel cognitive assessment.
- 3.1 Design Principle: Its 26 tasks target recognition, multimodal understanding, and high-level reasoning across varied real-world contexts.
- 3.2 Data Collection & Curation: 1,662 video segments are selected from roughly 8,000 candidates using taxonomy filters, audio-visual correlation and dynamics metrics, and human expert review.
- 3.3 Annotation Protocol: 80 professional annotators create questions requiring multimodal integration, while experts assess clarity, multimodal necessity, and difficulty.
- 3.3 Annotation Protocol: Automated verification uses vision-language and multimodal MLLMs to test modality requirements and flag questions that are too simple.
- 3.4 Dataset Statistics: The benchmark contains synchronized audio across 8 categories and 67 subcategories, averaging 141.1 seconds, with multiple-choice questions covering three cognitive levels.
- 3.4 Dataset Statistics: WorldSense distinguishes itself from static-image or weak-correlation benchmarks through open-domain videos, multi-task evaluation, and questions requiring audio-visual integration.
- 3.5 Evaluation Methodology: Each test instance pairs a synchronized video clip with a multiple-choice question, and performance is measured by accuracy against ground-truth answers.
4 Experiments and Findings
WorldSense evaluations expose substantial weaknesses in real-world omni-modal understanding, especially audio comprehension, cross-modal integration, and complex reasoning. Ablations show that visual information and original audio can improve performance when models integrate them effectively.
- Main Results: 54.0% is the maximum performance of open-source video models restricted to visual information.The comparison highlights the limitations of relying solely on visual processing for complex multimodal tasks.
- Main Results: Audio-visual models can underperform video-only models, indicating difficulty integrating both modalities effectively.Access to multiple modalities does not guarantee better performance without sophisticated integration mechanisms.
- Breakdown Results: Models consistently struggle with audio-related, spatial reasoning, counting, and emotion-related tasks.Emotion tasks require combining facial expressions, vocal tones, and contextual speech content.
- Breakdown Results: Performance varies across audio types, with event-related questions particularly difficult for Gemini 1.5 Pro compared with speech or music tasks.Other models also show inconsistent performance across audio types.
- Ablation Studies: Gemini 1.5 Pro improves from 34.6% with audio-only input to 48.0% with audio plus video.Visual information generally helps, although gains vary across models and captions can sometimes degrade performance.
- Ablation Studies: For Gemini 1.5 Pro, accuracy rises from 34.4% video-only to 39.3% with subtitles and 48.0% with original audio.Original audio contributes acoustic information beyond subtitles, including tone, emotion, and environmental sounds.
- Failure Analysis: Manual review of 130 Gemini 1.5 Pro samples identifies audio-understanding, visual-understanding, and reasoning errors.Most errors stem from audio deficiencies and reasoning failures.
5 Conclusion
WorldSense is introduced as a benchmark for evaluating MLLMs’ omni-modal understanding in diverse real-world contexts. Experiments expose substantial limitations in processing and coherently integrating visual and acoustic information.
- Conclusion: WorldSense evaluates joint visual-acoustic comprehension across diverse real-world contexts using curated question-answer pairs.The benchmark is intended to support evaluation of human-like omni-modal understanding.
- Conclusion: Experiments reveal significant limitations in current MLLMs’ ability to process and coherently integrate omni-modal information.The analysis demonstrates the importance of omni-modal collaboration in real-world understanding.
A Quality Control
WorldSense uses trained professional annotators, detailed instructions, and review procedures to support annotation quality. Training continued until annotators reached high proficiency, and similar QA pairs were revised.
- Annotator Preparation: 80 professional annotators with QA annotation experience formed the annotation team.The annotators were proficient in English and had participated in several QA annotation projects.
- Annotator Preparation: A one-week training program used 200 excluded videos until only 10% required modifications.The training set was not included in the final benchmark.
- Annotation Instructions: Annotators received comprehensive guidance covering tasks, question formulation, QA creation, examples, and cross-modal inference.
- Quality Control: The review process revised similar QA pairs as part of rigorous quality control.The authors attribute annotation quality to professional annotators, training, detailed guidelines, and review.
B Implement Details
The evaluation follows model-specific official implementations where available and uses standardized video, subtitle, audio, and question prompts. Prompt experiments selected the template that consistently performed best for VideoLLaMA2.
- Implementation Details: Open-source models follow their official implementations and preprocessing pipelines, while proprietary models use model-specific video input procedures.GPT 4o and Claude 3.5 Sonnet receive 16 uniformly sampled frames; Gemini 1.5 Pro receives raw video uploads through its official API.
- Evaluation Prompt: The evaluation prompt combines whole video frames, optional subtitles or audio, and a multiple-choice question.The study evaluates video-only, video-with-subtitles, and video-with-audio configurations.
- Evaluation Prompt: Video-only prompts ask models to select one answer from four options using the video frames.The response format is restricted to the letter A, B, C, or D.
- Evaluation Prompt: Subtitle prompts provide video frames followed by the video’s listed subtitles and the multiple-choice question.The response format remains a single answer letter.
- Evaluation Prompt: Audio prompts provide video frames with corresponding audio before the four-option question.Models are instructed to respond only with the letter of the selected answer.
- Prompt Selection: The prompt template ultimately used for VideoLLaMA2 consistently outperforms the alternative tested prompts.The experiment compares Prompt 1, Prompt 2, and the paper’s prompt.
E Limitation
WorldSense advances multimodal evaluation but acknowledges that its multiple-choice format limits assessment of generative, open-ended, and adaptable real-world understanding.
- The multiple-choice format constrains assessment of models’ generative capabilities.
- Open-ended responses, explanations, adaptability, nuanced reasoning, and creative problem-solving are not adequately evaluated.
- The authors plan to add open-ended questions and expand the evaluation paradigm.
H Datasheets
The dataset was created to evaluate real-world omnimodal understanding using videos, captions, and question-answer pairs assembled through human curation and model-assisted quality control.
- Composition: WorldSense contains 1,662 videos and 3,172 question-answer pairs representing videos with captions and QA annotations.
- Composition: Each instance includes video, corresponding audio, a question, an answer, video categories, fine-grained capabilities, audio-content classes, and auto-generated subtitles.
- Collection Process: The authors and contractors collected the dataset in 2024, sampling videos from FineVideo and Music-AVQA.
- Collection Process: Humans proposed questions and answers, while Qwen2-VL, Video-LLaMA2, and OneLLM supported quality control.
- Preprocessing/cleaning/labeling: Videos were selected by predefined categories and clipped using visual-audio correlation and dynamic scores.
H.5 Uses
WorldSense has been used to evaluate real-world video question answering and can also assess video-understanding capabilities of vision-language models.
- Uses: The dataset has been used to evaluate video question answering in real-world settings.
- Uses: The datasheet reports no repository linking papers or systems that use the dataset.
- Uses: WorldSense can also evaluate the video-understanding capability of vision-language models.
H.6 Distribution
WorldSense is publicly distributed and hosted through the project webpage, GitHub, and Hugging Face under a noncommercial Creative Commons license.
- Distribution: The dataset will be made publicly available to third parties.
- Distribution: The dataset is hosted on the webpage, GitHub, and Hugging Face.
- Distribution: The benchmark is released under the CC BY-NC 4.0 license, with no reported third-party or export restrictions.
- Distribution: The authors support and maintain the dataset, announce updates on GitHub, and currently report no erratum.