PaperlayerView exact revision on arXiv

Source-linked AI summary

A Benchmark for Omni-Modal Reasoning in Long Videos

Mohammed Irfan Kurpath, Jaseel Muhammad Kaithakkodan, Jinxing Zhou, Sahal Shaji Mullappilly, Mohammad Almansoori, Noor Ahsan, Beknur Kalmakhanbet, Sambal Shikhar, Rishabh Lalla, Jean Lahoud, Mariette Awad, Fahad Shahbaz Khan, Salman Khan, Rao Muhammad Anwer, Hisham Cholakkal

arXiv:2512.16978v2cs.CV

TL;DR

Long-video benchmarks have often traded off temporal scale, modality coverage, open-ended interaction, and interpretable scoring. LongShOTBench combines long-form omni-modal questions with criterion-level rubrics, and LongShOTAgent provides a training-free evidence-seeking approach. Evaluation reports substantial remaining headroom, with LongShOTAgent reaching 66.64% overall.

  • Problem

    Existing benchmarks often use short clips, omit non-speech audio, constrain user queries, or provide limited diagnosis of model errors.

    Source: §1 · p. 2

  • Method

    LongShOTBench pairs intent-driven single- and multi-turn questions with weighted evidence-linked rubrics, while LongShOTAgent iteratively searches, refines, and verifies multimodal evidence.

    Source: §1 · p. 3, §3.4 · p. 6

  • Results

    LongShOTAgent reaches 66.64% overall, while current evaluated models remain far from saturating LongShOTBench.

    Source: §1 · p. 4

  • Takeaways & Limitations

    LongShOTBench provides a diagnostic testbed for evaluating perception, temporal reasoning, and omni-modal understanding in long videos.

    Source: §6 · p. 11

    Model-side sampling variance is not explicitly quantified because each model receives a single generation pass.

    Source: G Limitations · p. 37

Abstract

from arXiv · show

Long-form omni-modal video understanding requires integrating vision, speech, and ambient audio with coherent long-context reasoning. Existing video benchmarks often trade off temporal scale, modality coverage, open-ended interaction, and interpretable scoring. To address this gap, we introduce LongShOTBench, a long video understanding benchmark designed around three coupled goals: holistic omni-modal integration, intent-driven open-ended interaction, and rubric-level diagnosis. It builds single- and multi-turn questions from real viewing scenarios, with systematic tasks probing visual, speech, ambient-audio, temporal, and cross-modal reasoning. Each item includes a reference answer and a weighted criterion-level rubric, letting evaluation identify which perceptual facts, temporal links, modality-grounding requirements, and reasoning steps are satisfied or missed. All samples are manually verified to improve grounding, clarity, and rubric reliability. We also introduce LongShOTAgent, a training-free omni-modal evidence-seeking agent coupling full-video preprocessing with targeted retrieval, query-adaptive segment refinement, and explicit claim verification over visual, speech, and non-speech audio evidence. Its iterative search-refine-verify loop exposes intermediate evidence and lets modality-specific specialists re-analyze relevant moments before answering. We evaluate 105 video-capable models spanning open-source omni-modal models, vision-language systems, audio LLMs, agentic pipelines and closed-source APIs. Current MLLMs remain far from saturating LongShOTBench, while our LongShOTAgent is the strongest training-free system, reaching 66.64% overall. By releasing the benchmark, leaderboard, and method, we provide a shared, interpretable testbed for advancing long-form omni-modal video reasoning. Code, data, and the leaderboard are available at https://longshot.cvmbzuai.com/.

1 Introduction

Long-video benchmarks have not combined extended temporal contexts, full vision-speech-audio coverage, open-ended intent-driven interaction, and interpretable scoring. LongShOTBench addresses these gaps with rubric-guided questions and a training-free evidence-seeking agent, while evaluation reveals substantial remaining headroom.

1 Introduction

  • Earlier benchmarks often use short clips, omit non-speech audio, restrict queries to templates or single turns, and provide limited diagnosis of answer failures.Longer duration alone has not resolved the modality gap, while richer audio-visual-language benchmarks tend to use shorter clips or narrow tasks.Source: §1 · p. 2
  • Removing or reordering multiple-choice options lowers strong monolithic models’ accuracy by more than 20% on Video-MME and WorldSense.The authors interpret this as evidence of reliance on option-level shortcuts rather than genuine understanding.Source: §1 · p. 3
  • LongShOTBench pairs long-form vision, speech, and non-speech audio with intent-driven single- and multi-turn questions and weighted, criterion-level rubrics.The rubric decomposes answers into verifiable components such as completeness, temporal localization, and cross-modal consistency, supporting partial credit and evidence-linked scores.Source: §1 · p. 3
  • LongShOTAgent preprocesses visual, speech, and audio signals, then iteratively retrieves relevant moments, refines segments with specialists, and verifies claims before answering.It is presented as a training-free reference point for structured omni-modal analysis of hour-scale video.Source: §1 · p. 3
  • 105 video-capable models are evaluated across closed-source APIs, open-source systems, and agentic methods, with substantial headroom reported across the board.The supplied passage states that Gemini 3.1 Pro Preview reaches 55.63% and begins describing the best open-source system.Source: §1 · p. 3
  • 66.64% is reached by LongShOTAgent, while Qwen3-Omni-30B-A3B-Thinking reaches 64.05%; rubric scoring also aligns with expert judgments across 400 criterion decisions.The three-verifier ensemble achieves κ = 0.75 and r = 0.92 against human consensus, with model rankings preserved up to near-tied adjacent pairs.Source: §1 · p. 4

2 Related Work

Prior long-video benchmarks generally lack the combination of full omni-modal coverage, long temporal contexts, higher-order reasoning, and intent-driven open-ended evaluation. Existing agentic approaches also focus predominantly on visual evidence and ASR transcripts.

Long-Video Benchmarks

  • LongShOTBench combines long contexts, vision, speech, and audio with intent-aware single- and multi-turn questions and weighted criterion-level rubrics.The cited comparison describes it as the only benchmark pairing full omni-modal coverage with open-ended, multi-turn, intent-driven evaluation and weighted rubrics.Source: §2 · p. 4

Agentic Video Understanding

  • VideoMind, Vgent, and Video-RAG operate predominantly over visual input and available ASR, and each scores below 15% on LongShOTBench.The authors state that these results indicate visual-centric agentic recipes do not yet generalize to hour-long omni-modal scenarios.Source: §2 · p. 4

3 LongShOTBench

LongShOTBench is constructed through sequential video processing, intent-based question design, reference-answer generation, weighted rubric scoring, and human validation. The resulting dataset contains long-form videos and both single- and multi-turn evaluation samples.

3 LongShOTBench

  • The dataset contains 274 long-form videos averaging about 41 minutes and 3,401 evaluation samples with 4,893 question-answer turns.The videos total about 188 hours, with roughly 12 samples and 18 questions per video.Source: §3 · p. 4

3.1 Omni-modal Caption Generation

  • Speech, visual scenes, and non-speech audio are processed in parallel at segment level, then merged into aligned cross-modal narratives.Speech activity defines segments; representative frames support dense visual captions, and the Figure 2 pipeline describes fusion before scenario and task mapping.Source: §3.1 · p. 4, Figure 2 · p. 5

3.2 Question Design

  • Questions are generated from practical user-intent scenarios and mapped to four categories, spanning perception, information, multimodal integration, and reasoning.The questions include single-turn queries and multi-turn dialogues, with difficulty distributed across five levels and most questions at moderate-to-hard levels.Source: §3.2 · p. 5

3.3 Answer Generation

  • Reference answers are generated from the captions and questions, with instructions to match detail to complexity, integrate relevant modalities, and abstain when metadata is insufficient.Answers are intended to remain grounded and concise rather than speculate about unobservable states.Source: §3.3 · p. 5

3.4 Hierarchical Rubric Generation

  • Each sample is scored by weighted, independently verifiable criteria checked as binary matches against a human-validated reference answer.This supports partial credit and links scores to specific evidence rather than asking verifiers for direct numeric judgments.Source: §3.4 · pp. 5–6

3.5 Human Validation

  • Ten trained annotators review samples, discarding unsuitable videos and revising questions, answers, and rubric criteria for clarity and factual grounding.The validation effort exceeded 450 annotation hours.Source: §3.5 · p. 6

4 LongShOTAgent

LongShOTAgent is a training-free baseline that preprocesses videos into a reusable multimodal store and uses an orchestrator to search, refine, and verify evidence before answering.

4 LongShOTAgent

  • A per-video vector database stores embedded frames, speech transcripts, and non-speech audio descriptions for query-time retrieval.Visual frames use SigLIP embeddings, while speech is transcribed with Whisper.Source: §4 · p. 6
  • The orchestrator iterates among semantic retrieval, specialist re-analysis of short clips, and claim verification until evidence is sufficient or a limit is reached.A final answer is synthesized from the retrieved evidence; the approach is presented as a reference point for LongShOTBench.Source: §4 · p. 6
  • The agent’s design uses modality-specific tools for visual content and for speech and non-speech audio.The authors state that this all-modality loop is more important than model scale alone for hour-length omni-modal questions.Source: §4 · p. 6

5 Experiments

The experiments benchmark 105 video-capable models and assess their scores with a weighted rubric and three-verifier ensemble. LongShOTAgent leads the benchmark, while results show persistent modality and perception challenges and verifier scores align closely with human judgments.

5.2 Benchmark Results

  • 66.64% overall puts training-free LongShOTAgent ahead of all 105 evaluated models; the benchmark remains far from saturated.Gemini 3.1 Pro Preview scores 55.63%, while Qwen3-Omni 30B-T scores 64.05%.Source: Table 2 · p. 7, §5.2 · p. 8

5.1 Evaluation Pipeline

  • The evaluation covers 105 video-capable models and scores criterion-level matches with three independent verifiers, averaging their scores for the overall result.Category scores are unweighted means of task scores, and the overall score is the unweighted mean of four category scores.Source: §5.1 · pp. 7–8, §5.2 · p. 8

5.2 Benchmark Results

  • The 30B Qwen3-Omni-Thinking scores 64.05%, versus 47.23% for the 235B Qwen3-VL-Thinking, indicating architecture matters more than scale in these comparisons.Omnimodal models with explicit audio pathways consistently outperform video-native VLMs of comparable or larger size, which lead frame-based systems.Source: §5.2 · p. 8
  • Top systems score higher on Reasoning than Core Perception: Qwen3-Omni 30B-T reaches 72.86% on RE and 51.44% on CP.The authors identify precise grounding of facts at specific moments as a priority for improvement.Source: §5.2 · p. 8
  • Non-speech audio trails visual performance: Qwen3-Omni 30B-T scores 62.09% on visual content and 55.89% on audio.LongShOTAgent shows the same pattern, with visual 64.96% versus audio 57.61%; timestamp- and stream-grounded non-speech events remain an open problem.Source: §5.2 · p. 8
  • Thinking variants outperform instruction-tuned siblings: Qwen3-Omni 30B-T scores 64.05% versus 39.17% for Qwen3-Omni 30B-I.Kimi-VL-T scores 24.32%, roughly 1.5× Kimi-VL-I at 15.76%.Source: §5.2 · p. 9
  • All three orchestrators benefit from LongShOTAgent, with gains of +12.95 to +38.52 points; on options-stripped Video-MME and WorldSense, the agent drops only 1.3 and 3.2 points and leads every open-ended column.Monolithic models lose 13 to 20 points when option-level shortcuts are removed; the agent generates answers from retrieved evidence.Source: §5.2 · p. 10, Figure 5 · p. 10, Table 3 · p. 10

5.3 Verifier-Human Alignment

  • Against human consensus, the verifier ensemble reaches κ=0.75 and F1=0.85 on criterion decisions, with sample-level Pearson r=0.92.The three-verifier mean preserves human system rankings up to two near-tied adjacent pairs.Source: §5.3 · pp. 10–11, Table 4 · p. 11

6 Conclusion

The paper presents LongShOTBench for diagnostic evaluation of long-form omni-modal video understanding and LongShOTAgent as a training-free approach to searching, refining, and verifying evidence. Evaluation of 105 video-capable MLLMs reveals substantial gaps in coherent reasoning over extended temporal contexts.

6 Conclusion

  • LongShOTBench evaluates MLLMs on long-form omni-modal video understanding across hour-long contexts, with open-ended intent-driven questions and rubric-guided diagnostics.The benchmark integrates vision, speech, and audio and covers perception, temporal reasoning, and omni-modal understanding.Source: §6 · p. 11
  • LongShOTAgent is a training-free framework that iteratively searches, refines, and verifies omni-modal evidence in long videos.Source: §6 · p. 11
  • Evaluation of 105 video-capable MLLMs reveals substantial gaps in current systems’ coherent omni-modal reasoning over extended temporal contexts.The paper presents the benchmark as a reproducible testbed for future model development.Source: §6 · p. 11

A Full Benchmark Results

The full leaderboard reports results for 105 evaluated video-capable models and LongShOTAgent, across six paradigms. Companion tables organize scores by category and judge, then by modality and video duration.

A Full Benchmark Results

  • Tables 5 and 6 report results for 105 evaluated video-capable models plus LongShOTAgent, for 106 leaderboard entries across six model paradigms.The paradigms include video-native and frame-based VLMs, native video-and-audio omni-modal models, audio-only LLMs, agentic systems, and a proprietary API.Source: A Full Benchmark Results · p. 18
  • Table 5 presents category and per-judge scores, with category values averaged across three judges and an overall average across judges.Its categories are Core Perception, Reasoning, Information, and Multimodal.Source: Table 5 · p. 18
  • Table 6 breaks out three-judge mean scores by modality and video-duration bucket.Modality scores use questions tagged Visual, non-speech Audio, or Speech; duration buckets are short (<20 min), medium (20–45 min), and long (>45 min).Source: Table 6 · p. 21

B.1 Dataset Statistics

LongShOTBench contains 274 long-form videos and 3,401 samples with 4,893 question-answer turns, totaling about 188 hours. Its source videos span varied genres, while its duration distribution centers around roughly 41 minutes.

B.1 Dataset Statistics

  • The dataset contains 274 videos, 3,401 samples, and 4,893 question-answer turns, totaling about 188 hours of video.Source: §B.1 · p. 25
  • Video durations range from 7.8 to 89.7 minutes, with a 41.3-minute mean and 40.9-minute median.Most videos lie between 30 and 60 minutes; the duration distribution has modes around 33–35 and 43–45 minutes.Source: §B.1 · p. 25, Figure 6 · p. 25
  • Documentary, Educational, and Lifestyle videos account for 67%, 55%, and 37% of the 274-video sample, respectively.Videos can belong to multiple genres, so these percentages do not sum to 100%.Source: Figure 7 · p. 26

C Human Alignment Study (Extended)

The extended human-alignment study compares verifier decisions with four expert annotators on 400 rubric criteria. The three-verifier mean tracks human scores and rankings closely, while the study notes limits arising from its small annotator pool and matched-input design.

C Human Alignment Study (Extended)

  • Four expert annotators independently graded 400 atomic rubric decisions drawn across four task categories, three duration buckets, and five representative systems.Human and verifier judgments use the same inputs and assess candidate-to-gold match for each criterion.Source: C Human Alignment Study (Extended) · p. 26
  • The study evaluates criterion-level agreement and F1 alongside whether verifier and human scores produce the same ranking of five systems.The three-verifier ensemble uses criterion-level majority voting; Cohen’s κ measures chance-corrected agreement on binary criterion labels.Source: C Human Alignment Study (Extended) · pp. 26–27
  • Across difficulty levels, average modality count rises from 1.44 at Level 1 to 2.47 at Level 5, while 76.4% of questions fall in Levels 3–5.Tri-modal questions reach 42.3% and 46.7% at Levels 4 and 5; reasoning questions peak at Level 4.Source: Figure 8 · p. 27, Figure 9 · p. 28
  • The three-verifier mean tracks human mean scores within about 0.1 across five systems and preserves rankings except for near-tied adjacent pairs (Spearman ρ=0.800).The two ranking flips involve verifier-side gaps below 0.04.Source: Table 7 · p. 27, C Human Alignment Study (Extended) · p. 27
  • Pairwise verifier κ ranges from 0.75 to 0.77, within the human-pair range of 0.53 to 0.82; the strongest verifier-human pair reaches κ=0.85.No verifier is an outlier in the pairwise agreement matrix.Source: C Human Alignment Study (Extended) · p. 28, Figure 10 · p. 29
  • The ensemble improves agreement over the best individual verifier (κ 0.74 to 0.75; F1 0.84 to 0.85), while individual verifiers have identical human rank-correlation of ρ=0.90.The authors justify the ensemble for absolute calibration and retain individual verifiers as a fallback for cost-constrained ablations.Source: C Human Alignment Study (Extended) · p. 28
  • The dense-human comparison uses only four annotators, and annotators judge match to the gold answer rather than world-knowledge correctness.The matched-input setup isolates rubric-verification fidelity; video grounding is handled upstream during dataset construction.Source: C Human Alignment Study (Extended) · pp. 28–29

D Detailed Benchmark Analysis

The additional analyses show that architecture, genre, model generation, and interaction format distinguish long-video performance in ways raw parameter count alone does not capture. Sparse architectures and benchmark difficulty vary substantially across model families and video genres, while single- and multi-turn scores are similar on average under the ideal-trajectory protocol.

Scaling Laws vs. Architectures

  • 64.05%: Qwen3-Omni 30B-T, activating 3B parameters, outperforms the 235B Qwen3-VL 235B-T at 47.23%, while Gemma-3 27B at 41.55% rivals InternVL3.5 241B at 29.77%.The analysis attributes the contrast to architecture and modality coverage rather than raw scale alone.Source: D Detailed Benchmark Analysis · p. 29

Parameter Efficiency: Dense vs. Sparse Architectures

  • 35.4% vs. 27.0%: open-weight MoE models average higher than dense models, and the MoE advantage is especially pronounced with thinking-mode training.The figure compares activated parameter count with overall score; dense scaling shows diminishing returns beyond approximately 30B parameters.Source: D Detailed Benchmark Analysis · p. 30, Figure 11 · p. 30

Best Models per Parameter Tier

Generational Progress Across Model Families

  • 8.5 points: Qwen3-VL 8B-I improves over Qwen2.5-VL 7B-I, while Gemma-4 scores below Gemma-3 at both tested scales.InternVL also improves from version 2.5 to 3.0: 4.9 points for the 38B variant and 3.7 points for the 78B variant; Figure 12 compares successive generations at matched scales.Source: D Detailed Benchmark Analysis · p. 31, Figure 12 · p. 32

Content Genre as a Difficulty Axis

  • 38.8% vs. 17.3%: Surveillance is the easiest genre and Comedy the hardest, a spread exceeding 20 points across genres.The analysis describes humor, fast editing, and narrative-bearing ambient audio as challenges in difficult genres.Source: D Detailed Benchmark Analysis · p. 31

Conversational Robustness: Single-Turn vs. Multi-Turn

  • 22.8% vs. 24.3%: mean single-turn and multi-turn scores differ by less than two points across the 105 models.The multi-turn protocol supplies the gold response from the previous turn as context, isolating per-turn response quality from error accumulation; top-tier models perform consistently across formats.Source: D Detailed Benchmark Analysis · p. 32

E MCQ Robustness Analysis

Across the examined multiple-choice benchmarks, monolithic models often lose substantial accuracy when option cues are removed or perturbed, while LongShOTAgent is comparatively robust. The results illustrate how open-ended and adversarial protocols reveal performance differences that standard multiple-choice accuracy can obscure.

Evaluation Protocols

  • The evaluation compares canonical four-option MCQ with open-ended answering, shuffled options, and original options plus NOTA/AOTA distractors.For open-ended scoring, models first answer without options, then select the matching choice; the gold answer remains among the original four options in the NOTA/AOTA protocol.Source: E MCQ Robustness Analysis · pp. 32–33

Open-Ended Evaluation

  • 69.91%: LongShOTAgent’s open-ended Video-MME score drops only 1.3 points from its standard-MCQ score, while Qwen3.5-35B-A3B drops 20.5 points to 58.4%.LongShOTAgent has the smallest open-ended degradation among the evaluated systems; Table 9 summarizes accuracy across four protocols and deltas from standard MCQ.Source: E MCQ Robustness Analysis · pp. 33–34, Table 9 · p. 35

Option Perturbations

  • 68.11%: LongShOTAgent leads the Video-MME shuffled-options comparison, dropping 3.1 points; under Same+NOTA it scores 66.09%, down 5.2 points.Qwen3-Omni-Instruct falls to 33.19% under shuffling and 29.67% under Same+NOTA.Source: E MCQ Robustness Analysis · p. 34, Table 9 · p. 35

WorldSense

  • 43.38%: LongShOTAgent leads open-ended WorldSense, after the Omni models’ standard-MCQ advantage disappears.On shuffled WorldSense, it is the only method whose score rises, from 46.56% to 50.70%; its modular audio path closes most of the native-audio gap when option cues are removed.Source: E MCQ Robustness Analysis · pp. 34–35, Table 9 · p. 35

Implications for Evaluation

  • MCQ accuracy alone is a poor proxy for video understanding, and LongShOTAgent’s 66.64% LongShOTBench score is consistent with its advantage under evidence-grounded evaluation.The authors report that the agent outperforms the monolithic alternatives they evaluated under LongShOTBench’s rubric-graded, free-form protocol.Source: E MCQ Robustness Analysis · p. 35

F Additional Details of LongShOTAgent

LongShOTAgent is a training-free system that indexes video evidence and uses an orchestrator to retrieve, refine, and verify relevant material. Its reported performance is presented as an example of modular coordination under the benchmark’s video-only and computational constraints.

Motivation & Advantages

  • A semantic index retrieves and refines only prompt-relevant clips, while an explicit search-refine-verify loop gathers evidence before the final answer.The method describes segment refinement on windows of up to 60 seconds and claim verification against the visual scene.Source: F Additional Details of LongShOTAgent · pp. 35–36

Domain Specialization

  • The pipeline assigns visual and acoustic work to specialized modules, with an LLM orchestrator directing the query toward the relevant branch.The described components include a vision-language module and Audio Flamingo 3 for speech and non-speech audio; faster-whisper supplies timestamped transcripts during indexing.Source: F Additional Details of LongShOTAgent · pp. 35–36

Motivation & Advantages

  • 66.64%: LongShOTAgent outperforms the evaluated state-of-the-art open omni-models and proprietary APIs without additional training or private data.The system is described as a reference demonstration of structured orchestration, not part of the primary model benchmarking.Source: F Additional Details of LongShOTAgent · pp. 35–36

Execution Dynamics

  • The system uses a per-video multimodal index and an orchestrator that iteratively searches candidate moments, refines short segments, and verifies claims before answering.Indexing runs once per video; during reasoning, intermediate outputs remain in conversation history for refining hypotheses across rounds.Source: F Additional Details of LongShOTAgent · pp. 35–36

Fairness and Scope

  • Its fairness account treats selective activation and modular coordination, rather than summed component parameters, as the relevant efficiency measure.The authors say components are frozen and publicly available, and that the improvement over single end-to-end models arises from orchestration rather than additional data or tuning.Source: F Additional Details of LongShOTAgent · p. 36

G Limitations

The authors identify constraints in scoring, inference measurement, model-input comparability, dataset coverage, and benchmark construction. These boundaries qualify how the reported scores and benchmark coverage should be interpreted.

Scoring

  • Scores may retain verifier-specific bias and prompt sensitivity because rubric decisions rely on a three-model verifier ensemble.The evaluation grounds decisions against human-validated gold answers and averages verifier scores, but the authors note residual effects may influence absolute scores.Source: G Limitations · pp. 36–37

Inference Measurement

  • Model-side sampling variance is not explicitly quantified because the study uses one generation pass per model.The reported score is the mean of three verifiers’ overall scores; per-verifier values and inter-verifier standard deviation are in the supplement.Source: G Limitations · p. 37

Model-Input Comparability

  • The native video-only protocol excludes commercial APIs requiring client-side frame extraction, preventing direct comparison under identical conditions.The authors cite GPT-4o as an example.Source: G Limitations · p. 37

Dataset and Construction Scope

  • The dataset is predominantly English and sourced from Video-MME and YouTube, limiting coverage of other languages, cultures, and low-resource domains.The question and rubric construction pipeline also relies on a single LLM family, Qwen3; human validation is intended to mitigate, but not fully eliminate, propagated biases.Source: G Limitations · p. 37
Loading 2512.16978v2…