OmniBench: Towards The Future of Universal Omni-Language Models

1 Introduction

• Existing benchmarks often assess single modalities or image–text and audio–text pairs, leaving comprehensive evaluation of simultaneous tri-modal integration lacking.The introduction frames this gap as limiting the community’s ability to assess and improve holistic model capabilities.Source: §1 · pp. 1–2
• OmniBench tests recognition, interpretation, and reasoning across visual, acoustic, and textual inputs, with correct answers requiring integrated multimodal understanding.The benchmark relies on human annotations and includes annotator-provided answer rationales.Source: §1 · p. 2
• Open-source OLMs often exceed random-guess accuracy but can struggle to follow instructions with image and audio together; proprietary models perform better overall but lose more accuracy when either modality is ablated.Human evaluators perform better overall than models, with a distinct distribution of strengths; VLMs and ALMs using textual alternatives remain at a preliminary level of tri-modal understanding.Source: §1 · pp. 2–3

2 Related Work

• Vision-language benchmarks cover capabilities such as OCR, spatial awareness, retrieval, and reasoning, while audio benchmarks target speech recognition, audio QA, and sound classification.These benchmark families primarily evaluate their respective modality domains or paired inputs.Source: §2 · p. 3
• The related-work section identifies a significant gap in benchmarks that assess simultaneous integration of textual, audio, and visual inputs.It defines OLMs as models capable of processing at least three modalities simultaneously.Source: §2 · p. 3
• Existing audio-visual QA datasets often emphasize object and sound identification, with limited causal inference, abstract reasoning, or instruction-following evaluation.Some tasks can be answered from a single modality, despite efforts to improve human annotation and alignment.Source: §2 · p. 3

3 OmniBench

• OmniBench targets concurrent image, audio, and text inputs, beginning with these well-explored modalities while leaving the broader “omni” concept extendable.The benchmark contains 1,142 questions across task types and audio categories, including speech, sound events, and music.Source: §3 · p. 3, §3.1 · pp. 4–5

3.1 Benchmark Design

• Its taxonomy spans spatial-temporal entities, causal inference, and abstract concepts, with questions probing perceptual, symbolic, causal, quantitative, and commonsense reasoning.The benchmark avoids explicit chain-of-thought traces and includes spatial-imagery reasoning such as interpreting soundstage cues.Source: §3.1 · pp. 4–5

3.2 Annotation Protocol

• Annotations must require both image and audio information; human cross-inspection and MLLM checks reject samples solvable from limited inputs, while 9.58% (121) QA pairs were dumped as hard to recycle.Questions are multiple-choice, with confusing wrong options, and annotators provide transcripts or music captions and answer rationales.Source: §3.2 · p. 5

3.3 OmniInstruct

• OmniInstruct is built from MSRVTT-QA, AVQA, and Music-AVQA2.0 train and validation data, filtering multimodal QA samples for relevance, ambiguity, and dependence on both image and audio.The processing pipeline avoids the first and last five video frames and retains 96k samples for training and validation.Source: §3.3 · p. 6

4 Experiment Settings

• Baselines include omni-language, vision-language, and audio-language models, proprietary API systems, and a human-expert baseline from three musicians.The authors do not classify proprietary systems as VLMs, ALMs, or OLMs because their mechanisms are undisclosed.Source: §4 · p. 6
• Accuracy on four-option questions is the main metric, with image and audio ablations used to assess reasoning and benchmark robustness.Random guessing yields 25% accuracy.Source: §4 · p. 6, Table 3 · p. 7
• The proprietary-model classification reflects an investigation conducted on September 22, 2024.This date qualifies the authors’ decision not to assign those systems to a model family.Source: §4 · p. 6
• For vision-language models, human-annotated audio transcripts substitute for audio; for audio-language models, InternVL-2-76B-generated image captions substitute for images.Models use the same inference process and metric calculation as in the vanilla setting.Source: §4 · p. 7

5.1 Results on Omni-Language Models

• Most open-source OLM baselines exceed random guessing, Qwen-2.5-Omni can outperform proprietary models, and Gemini-1.5-Pro remains below 50% accuracy.Gemini-1.5-Pro’s full-input and ablated-setting comparison suggests that it uses information from all modalities.Source: §5.1 · p. 7
• Open-source baselines generally do better on speech; models such as Gemini-1.5-Pro perform better on object identification than plot or story reasoning, while quantity and counting remain difficult.Scaling UnifiedIO contributes to quantity and counting performance.Source: §5.1 · p. 8
• Music results vary by model: Video-Salmonn and Gemini-1.5-Pro do better on music than speech, while music-model performance can reflect training-data bias and limited annotation.Qwen2-Audio performs strongly on MTG-Jamendo but generalizes poorly to MTT; music is more restricted by copyright and expensive to annotate than speech.Source: §5.1 · p. 8
• Human evaluators reached 63.19% accuracy, higher than all OLMs, with particularly strong results on Sound Event and Abstract Concept tasks.The three annotators’ results had a Fleiss’ Kappa of 0.421.Source: §5.1 · p. 8

5.2 The Effectiveness of OmniInstruct

• 29.2% OmniBench accuracy for voice-filtered MIO-Instruct exceeded 25.7% with vanilla fine-tuning and 24.8% for baseline MIO-Instruct.Using full OmniInstruct for LoRA fine-tuning improved MiniCPM-o-2.6 from 40.5% to 45.9%. The voice-filtered approach removes non-speech audio tokens to align with MIO-Instruct’s speech-only audio tokenizer.Source: §5.2 · p. 9

5.3 Textual Approximation on Images and Audios

• 6.42% average performance gains for UnifiedIO-2 models came from audio replacement with text, versus average drops of 1.87% for image replacement and 0.12% for replacing both.The authors interpret this pattern as indicating shortcomings in audio modeling and potential noise in generated image captions compared with human-written audio transcripts.Source: §5.3 · p. 9
• VLMs generally outperform ALMs when text substitutes for audio and images, though proprietary VLMs outperform open-source VLMs and results vary across audio types.Claude-3.5 and GPT-4o are generally the strongest VLMs; the best ALM varies by audio type.Source: §5.3 · p. 9
• 47.02% accuracy for Qwen2-Audio-7B-Instruct with both textual representations exceeded 39.05% with audio transcript only and 39.67% with image caption only.Most models improved with textual representations of both modalities, while GPT-4o (0513) reached 60.60% in that setting and its gain was not significant.Source: §5.3 · p. 10, Figure 4 · p. 10

• Open-source multimodal models struggle with simultaneous processing of visual, acoustic, and textual inputs.OmniBench is presented as a tool for guiding progress in multimodal language models.Source: §6 · p. 10
• Textual approximations reveal a general speech-audio bias and stronger vision-language model performance than audio-language models.Source: §6 · p. 10
• The authors call for improved multimodal integration, diverse training datasets, and techniques to reduce modality bias.They position OmniBench as a tool to guide future development.Source: §6 · p. 10

• Robust omni-language models could support healthcare diagnostics, assistive technologies, and more natural human-computer interaction.Source: Impact Statement · p. 11
• OmniBench evaluates integration of visual, acoustic, and textual information, while OmniInstruct offers a resource for training and fine-tuning models.Source: Impact Statement · p. 11
• The statement identifies bias and fairness as ethical considerations and says comprehensive evaluation can help researchers identify and address them.Source: Impact Statement · p. 11

A More Experiment Results

• Table 8 reports results with images and audios represented as text; its second and third columns use corresponding models from Tables 6 and 7.Source: Table 8 · p. 16

B.1 Statistics for OmniInstruct Dataset

• Only 96k samples remained for Omni-modality SFT training after most dataset samples were filtered out as low quality.The authors note that template-generated questions could be paired with images that were not relevant to the question.Source: §B.1 · p. 16

B.2 Discussion of the Human Annotation Protocol

• Sixteen annotators and five quality reviewers covered music, linguistics, phonetics, cryptography, and math.Annotators documented modality-specific rationales, which verifiers checked for consistency.Source: §B.2 · p. 16

B.3 Diversity of Music Audios of OmniBenchmark

• The music subset spans global traditions and genres, from classical, jazz, and avant-garde music to popular, folk, and experimental works.Examples include Kunqu opera, music from India, the Arab world, Africa, and Japan, and Asian folk oral traditions.Source: §B.3 · pp. 16–17

B.4 The Discussion of Video Data

• The benchmark focuses on static images and audio to study targeted cross-modal reasoning rather than video’s richer temporal dynamics.The authors say limited reliable video-plus-audio support among models at development time made that setting difficult to include.Source: §B.4 · p. 17

B.5 Prompt for Quality Control on OmniInstruct Dataset

• Figure 5 presents the prompt used for quality control of the OmniInstruct dataset.Source: Figure 5 · p. 17

B.6 Audio Transcription and Caption

• Speech and song use transcripts, while instrumental music and other non-speech audio receive expert-written descriptions of salient audio features.Music-based questions were manually vetted for answerability from image and audio transcriptions alone.Source: §B.6 · p. 17
• The authors did not test spectrogram-based VLM embeddings, noting that suitable spectrogram types and resolutions vary by task.They use expert-written audio summaries as semantically compact, modality-aligned representations accessible to current vision-language models.Source: §B.6 · p. 17

C Data Annotation Platform

• The platform supported structured editing, version control, and multi-stage review during annotation correction and quality inspection.Figure 6 shows a snapshot of the platform interface.Source: C Data Annotation Platform · p. 18
• Figure 6 provides a snapshot of the data annotation platform used for human evaluation.Source: Figure 6 · p. 18

D Statistic Significance of the Findings

• The authors used multiple types of hypothesis testing to examine whether observed findings could stem from a small test set.They report about 30 claims in the discussion section.Source: D Statistic Significance of the Findings · p. 18

D.1 Inference if Two Models Provide Significantly Different Results under the Same Small Subset

• The tests model each sample’s correctness as a binary outcome and address how small test sets can make estimated accuracy rankings misleading.The appendix describes a normal-law-based sample-size calculation for a target margin of error.Source: §D.1 · pp. 18–19
• 38.00% UnifiedIO2-xlarge exceeds xxlarge (33.98%) on 1,142 samples (p<0.01); Gemini-1.5-Pro with image and audio scores 42.91% versus 27.93% with image only (p=1.19e-20).Source: §D.1 · p. 19
• 59.37% Claude-3.5 exceeds InternVL-2-40B (54.29%), while GPT-4o (57.62%) exceeds InternVL-2-26B (51.75%) but not InternVL-2-40B.Source: §D.1 · p. 19
• 56.6% Video-SALMONN on music exceeds Gemini-1.5-Pro (46.23%, p<0.05), while Gemini-1.5-Pro leads other models on general audio (42.26%) and speech (42.67%) comparisons.Source: §D.1 · pp. 19–20
• 47.02% Qwen2-Audio-7B-Instruct with image caption and audio transcription exceeds its transcription-only (39.05%) and caption-only (39.67%) results (p<0.01).Claude-3.5-Sonnet and GPT4-o show no significant changes between the compared inputs.Source: §D.1 · p. 20

D.2 Comparasion between Two Different Setting with the Same Samples

• Paired t-tests compare UnifiedIO2 settings on the same samples; audio-transcription input differs from image-and-audio input (p=0.0471), while image-caption comparisons show no significant difference.Source: §D.2 · p. 20

D.3 Comparasion on the Same Setting with Two Different Subset Samples

• Gemini-1.5-Pro scored 32.91% on Plot Inference (237 samples), worse than the other seven classes (p=6.69e-4), and 62.56% on Object Identification & Description (211 samples), better than others (p=9.46e-11).Source: §D.3 · p. 21
• The authors note that 20–30 hypothesis tests can yield p-values below 0.05 by chance, though dependence among tests makes the observed significant results less indicative of false positives.Source: §D.3 · p. 21

E Human Evaluation

• 74.03% was the score from a second set of experts given 2–10 minutes, compared with the stricter human-evaluation time limit.The comparison was intended to estimate task difficulty versus ambiguity.Source: E Human Evaluation · p. 21
• Time constraints limited annotators’ reasoning, while background knowledge, language proficiency, and music expertise may also have affected human performance.Source: E Human Evaluation · p. 21

F Limitation

• High annotation costs—about 30 minutes to design and 10 minutes to verify each example—limit this version of OmniBench to 1,000 manually curated examples.The authors plan to expand the dataset in future iterations.Source: F Limitation · p. 21
• LLM-generated image captions can underperform human descriptions on precise visual text recognition, while the benchmark currently excludes video and temporal reasoning.The authors identify extending the benchmark to video and audio temporal reasoning as future work.Source: F Limitation · p. 22

G Experiments Compute Resources

• Inference on A800 GPUs took 30 minutes to 3 hours per model, public API latency introduced variability, and OmniInstruct LoRA fine-tuning took 5 minutes per epoch on eight H800 GPUs.The passage does not specify exact memory configurations.Source: G Experiments Compute Resources · p. 22

H Ethical Statement

• The ethics statement reports annotators were compensated according to local minimum-wage rules, MMAR audio clips came from public videos and were under 30 seconds without personal data, and release is CC-BY-NC noncommercial.It also describes balanced speech and vocal-music data across multiple spoken and sung languages.Source: H Ethical Statement · p. 22

NeurIPS Paper Checklist

• The checklist reports that benchmark data and evaluation code are public, test-time settings are documented, and Appendix D describes three types of hypothesis testing.It marks formal theory proofs as not applicable because the paper presents no theoretical results.Source: NeurIPS Paper Checklist · pp. 24–26

8. Experiments compute resources

• The checklist characterizes the work as posing no immediate societal-impact risks and says high-risk-release safeguards are unnecessary because it releases no high-risk models or data.Its justification cites the benchmark’s foundational research scope and noncommercial license.Source: §8 · pp. 27–28
• The authors report that data collection and annotation were conducted by authors, qualified collaborators, or company employees, with no direct participant interaction or personally identifiable information collected.The study was reviewed and approved by 01.ai’s Human and Artefacts Research Ethics Committee; the IRB question is marked NA.Source: §8 · p. 29
• LLMs assessed whether multiple-choice options were confusing, while VLMs generated image captions for ALM evaluation and generated and filtered OmniInstruct data.The paper declares these uses as important or non-standard components of the research.Source: §8 · p. 30