Source-linked AI summary
Video-Oasis: Rethinking Evaluation of Video Understanding
Geuntaek Lim, Sungjune Park, Jaeyun Lee, Inwoong Lee, Taeoh Kim, Dongyoon Wee, Minho Shim, Yukyung Choi
TL;DR
Video understanding benchmarks may conflate visual perception, linguistic reasoning, and knowledge priors, making reported Video-LLM performance difficult to interpret. Video-Oasis audits existing benchmarks with visual, temporal, and ambiguity diagnostics, finding that 55% of samples are shortcut-solvable and that filtered video-native challenges leave state-of-the-art models only marginally above random chance.
Problem
Video benchmark performance is difficult to interpret because it may reflect visual perception, linguistic reasoning, or knowledge priors rather than robust video understanding.
Method
Video-Oasis audits existing benchmarks using visual-dependency, temporal-dependency, and ambiguity diagnostics to distill video-native challenges.
Results
55% of existing benchmark samples are solvable without visual input or temporal context, while filtered video-native challenges yield state-of-the-art performance only marginally above random chance.
Takeaways & Limitations
The distilled challenges provide a testbed for studying algorithmic design choices for robust video understanding.
Takeaways & Limitations
Audio and Summary probes are diagnostic inputs rather than optimized text-only video-reasoning methods.
Abstract
from arXiv · showhide
The inherent complexity of video understanding makes it difficult to determine whether Video-LLM benchmark performance stems from visual perception, linguistic reasoning, or knowledge priors. While many benchmarks have emerged to assess high-level reasoning, shared criteria for evaluating video understanding remain largely overlooked. Instead of introducing yet another benchmark, we take a step back to re-examine the criteria for evaluating video understanding. In this work, we introduce Video-Oasis, a sustainable diagnostic suite for systematically auditing existing video understanding benchmarks. This audit reveals that 55\% of existing benchmark samples are solvable without visual input or temporal context. After filtering these shortcuts, the remaining video-native challenges expose a substantial capability gap: state-of-the-art models perform only marginally above random guessing. Building on these findings, we use the distilled challenges as a testbed to investigate which algorithmic design choices contribute to robust video understanding. We hope our work provides a practical foundation for constructing rigorous video benchmarks and evaluating future Video-LLMs. Code is available at https://github.com/sejong-rcv/Video-Oasis.
1 Introduction
Video-Oasis re-examines whether video benchmarks genuinely require visual and temporal understanding, auditing shortcuts and ambiguity across existing evaluations. The audit finds widespread shortcut solvability and only marginal model performance on the remaining video-native challenges.
- Video-LLM benchmarks increasingly combine fine-grained dynamics with long-form reasoning, complicating attribution of performance to perception, language, or knowledge priors.
- Video-Oasis audits whether benchmark samples require visual evidence, temporal context, and reliable annotations before distilling video-native challenges.Its tests remove or perturb visual evidence and temporal order, while ambiguity verification uses human inspection.
- Reported benchmark accuracy is strongly correlated with shortcut prevalence across 14 diverse benchmarks.The audit covers tasks from perception to reasoning and videos ranging from seconds to hours.
- 55% of existing benchmark samples are solvable without visual input or temporal context.
- After shortcut filtering, state-of-the-art models perform only marginally above random chance on video-native challenges.These challenges expose substantial limitations in current models’ video understanding capabilities.
- Video-Oasis jointly examines visual dependency, temporal dependency, and ambiguity, using cross-model consensus and human verification to improve diagnostic reliability.
2 Related Work
Video benchmark construction has become harder as Video-LLMs and datasets expand, while automatic pipelines may not test video-specific properties. Video-Oasis addresses this with diagnostics spanning visual, temporal, and ambiguity dimensions.
- Video-LLMs and agentic methods improve existing benchmark performance, increasing the need for rigorous evaluation that properly attributes these gains.
- Automatic dataset pipelines often generate questions from keyframes or transcripts, leaving unclear whether benchmarks evaluate video-specific properties.The cited properties include temporal continuity and causal interaction.
- Prior auditing work identifies temporal shortcuts, perception-oriented task concentration, and benchmark redundancy.
- Video-Oasis diagnoses benchmark quality through visual-dependency, temporal-dependency, and ambiguity assessments.
3 Video-Oasis: Diagnostic Suite for Video Understanding
Video-Oasis audits whether benchmark questions genuinely require visual evidence, temporal context, and reliable annotations. Its diagnostic tests reveal widespread shortcut solvability and validate the identified shortcuts across diverse benchmarks.
- Diagnostic criteria: Video-Oasis systematically tests visual dependency, temporal dependency, and annotation reliability in existing video benchmarks.The suite replaces or disrupts visual and temporal evidence and flags ambiguous, redundant, or potentially misclassified samples for inspection.
- Diagnostic criteria: Diagnostic tests include blind, audio, summary, center-frame, frame-shuffling, consistency, redundancy, and sensitivity checks.These probes distinguish linguistic or auditory shortcuts, static recognition, temporal insensitivity, and unreliable annotations.
- Benchmark audit results: 30% to 50% accuracy persists when raw visual evidence or temporal order is removed or disrupted, versus a 25.6% chance baseline.The aggregate results cover 14 benchmarks and indicate that many samples do not enforce intended visual or temporal dependencies.
- Benchmark audit results: Under consensus threshold c ≥1, an average of 92.7% of samples show shortcut-solvable behavior under at least one diagnostic test.Shortcut cases occur consistently across spatial, temporal, reasoning, and general benchmark groups.
- Diagnostic distribution: Summary, Center-Frame, and Frame Shuffling account for 65% of unique shortcuts, while Blind, Audio, and Bag-of-Frames account for 35%.Ambiguity tests refine the final set: consistency and redundancy address unreliable annotations, while sensitivity corrects frame-shuffling false positives.
- Validation: Shortcut-identified samples achieve a 76% average correlation rate under standard evaluation across tests and models.The validation uses models different from those used to identify shortcuts, supporting the reliability of the diagnostic process.
4 Distilling the Challenges of Video Understanding
Video-Oasis filters shortcut-solvable questions to distill video-native challenges from existing benchmarks. The resulting set targets diverse spatio-temporal capabilities and reveals substantial difficulty for current models.
- Challenge distillation: Metadata abstraction and model-assisted clustering consolidate the surviving tasks into five unified challenge categories.An ensemble of five proprietary LLMs accepts a category when at least three models agree; otherwise, the sample is manually inspected.
- Challenge categories: The five categories are Fine-Grained Perception, Spatial World Understanding, Temporal Dynamics & Tracking, Causality & Logical Reasoning, and Global Narrative.They cover evolving visual details, multi-view spatial inference, temporal changes, latent cause-effect relations, and long-horizon event integration.
- Distilled set: The distilled challenges require fine-grained perception, spatial understanding, temporal tracking, causal reasoning, and global narrative understanding.Examples move beyond frame-level recognition toward cross-view matching, chronological reasoning, and preserving temporal context.
- Challenge distillation: Video-Oasis uses surviving benchmark samples to identify video-native challenges rather than introducing new tasks or imposing predefined categories.The categories emerge through a bottom-up, data-driven filtering process.
- Distilled set: 11,033 QA pairs from 4,938 unique videos remain from 24,416 original pairs, reducing evaluation volume by 55%.The distilled set preserves essential spatio-temporal challenges while enabling more efficient, reproducible, and extensible evaluation.
- Challenge validity: The Video-Oasis-distilled set overlaps only 44.6% with a baseline of 6,609 questions all answered incorrectly by three current models.This comparison indicates that Video-Oasis identifies more than questions that are simply difficult for current models.
- Model evaluation: Current Video-LLMs perform close to the 25.6% chance level on the distilled challenges, except for a few top-tier models.Gemini-2.5-Pro achieves the highest performance across all skills, while Global Narrative remains a primary bottleneck.
5 Exploring Algorithmic Designs
The distilled challenges serve as a testbed for algorithmic choices in temporal grounding, reasoning-depth control, and training. Experiments identify grounding, adaptive reasoning, and complementary SFT/RLVR strengths as important design factors.
- Temporal grounding: Temporal grounding consistently improves model performance on the distilled video-native challenges.The ablation retrieves the 16 most relevant frames with AKS as a representative grounding method.
- Temporal grounding: Oracle grounding raises performance from 35.0% to 50.8% on 1,060 distilled samples, compared with 78.0% to 80.8% on 1,885 shortcut samples.The larger gain on distilled samples confirms that shortcuts more often permit bypassing precise temporal grounding.
- Reasoning depth: An oracle ensemble selecting between instruction-following and thinking modes reaches 46.2 accuracy.This result nearly closes the gap with frontier-level performance and motivates adaptive reasoning-depth selection.
- Training paradigms: Long-context SFT improves overall accuracy from 29.2% to 34.5% over the Qwen2.5-VL baseline without RLVR.The comparison uses Eagle2.5 and indicates that long-context optimization can enhance general spatio-temporal reasoning.
- Training paradigms: RLVR results vary non-linearly across Video-R1 (26.3%), Qwen2.5-VL (29.2%), and VideoAuto-R1Qwen2.5 (32.7%).The results identify reward formulation as an important factor in RL-based video training.
- Training paradigms: SFT and RLVR show complementary strengths rather than a single superior training path.Well-optimized SFT improves overall accuracy, whereas RLVR with grounding rewards produces larger gains on particular reasoning challenges.
- Implications: The algorithmic studies provide practical guidance for designing stronger video understanding methods.The paper highlights temporal grounding and adaptive reasoning as primary drivers of spatio-temporal reasoning.
6 Conclusion
Video-Oasis provides a diagnostic lens for benchmark vulnerabilities and uses the resulting challenges to study robust video understanding. The paper highlights temporal grounding, adaptive reasoning, and training balance as key algorithmic considerations.
- Conclusion: Video-Oasis diagnoses benchmark vulnerabilities in visual and temporal dependency while re-examining the current video-understanding landscape.Its diagnostic suite is designed to support robust video understanding evaluation.
- Conclusion: Temporal grounding and adaptive reasoning are identified as primary drivers of spatio-temporal reasoning.The conclusion summarizes the algorithmic exploration conducted on the distilled challenges.
- Conclusion: The paper further identifies the balance between training approaches as an important consideration for video understanding.This conclusion follows the paper’s broader algorithmic analysis of training paradigms.
Video Understanding (Supplementary Material)
The supplementary material provides robustness analyses, benchmark-wise filtering and diagnostic results, implementation details, challenge statistics, and reproducibility experiments.
- Section A analyzes Video-Oasis robustness under alternative diagnostic model configurations.
- Section B reports benchmark-wise shortcut-filtering outcomes and diagnostic test results.
- Section C documents Video-Oasis implementation details, while Section D presents statistics, qualitative examples, and annotation prompts for distilled video-native challenges.
- Section E adds experimental results related to reproducibility.
A Robustness of Video-Oasis
Video-Oasis remains robust across alternative diagnostic model configurations, with shortcut sets closely matching those from the main configuration.
- Alternative language-only models and video backbones are used to repeat visual and temporal dependency diagnostics.
- Shortcut samples are defined only when all three diagnostic models answer correctly.
- 87% is the minimum overlap between shortcut sets from alternative and main diagnostic model configurations.All reported overlaps exceed 87%.
- The supplementary analysis includes quantitative diagnostic results and shortcut-set overlap tables, alongside benchmark-wise analyses.
B.1 Filtering Results of Video-Oasis
Video-Oasis filters benchmark samples using shortcut diagnostics, with benchmark-specific filtering ratios and performance gaps that do not follow a simple difficulty pattern.
- B.1 Filtering Results of Video-Oasis: Table S3 reports original and remaining QA-sample counts, filtering ratios, and Qwen2.5-VL (7B) performance gaps for each benchmark.
- B.1 Filtering Results of Video-Oasis: 76.4 versus 66.9 is the EgoSchema versus MVBench filtering-ratio comparison, despite MVBench having higher original accuracy.EgoSchema also has the larger performance gap: 40.5 versus 20.7.
- B.1 Filtering Results of Video-Oasis: 40.5 versus 20.7 is the EgoSchema versus MVBench performance-gap comparison under Qwen2.5-VL (7B).
- B.1 Filtering Results of Video-Oasis: The supplementary benchmark-wise analysis also reports diagnostic results for Blind, Audio, Summary, Center-Frame, Frame Shuffling, and Bag-of-Frames tests.Audio evaluation excludes videos lacking usable audio tracks or speech.
- B.1 Filtering Results of Video-Oasis: Bag-of-Frames benchmark-wise results are provided separately for frames processed independently without temporal modeling.
C Implementation Details of Video-Oasis
Video-Oasis evaluates visual and temporal dependency by replacing, abstracting, or perturbing video inputs, then applying model-specific diagnostic procedures and annotation checks.
- C.1 Visual Dependency Tests: Visual dependency tests remove raw visual evidence using Blind, Audio, and Summary configurations.These replace the video with no auxiliary input, a speech transcript, or concatenated captions, respectively.
- C.1 Visual Dependency Tests: V denotes visual input, while Pdiag formats diagnostic context with the question and answer options before the model responds.
- C.2 Temporal Dependency Tests: Temporal dependency tests disrupt frame selection or ordering through Center-Frame, Frame Shuffling, and Top-k Matching strategies.Top-k Matching retrieves frames most similar to the question; the other strategies use a center frame or randomly permuted frames.
- C.2 Temporal Dependency Tests: The temporal diagnostic procedure returns responses for center-frame, shuffled-frame, and top-k frame subsets.
- C.2 Temporal Dependency Tests: MLLMs process configured temporal subsets for temporal reasoning, whereas VLMs process frames independently and use top-k cosine-similarity matching.
- Annotation Diagnostics: Consistency, Redundancy, and Sensitivity tests identify ambiguity, weak temporal constraints, and false positives from temporal perturbations.Sensitivity inspection restores samples that still require temporal ordering despite shuffled-input success.
D.1 Video-Native Challenges Statistics
Video-Oasis organizes video-native challenges across five categories and diverse video durations, with temporal dynamics and tracking forming the largest category. The suite also reveals answer-format and benchmark-specific differences that support a unified evaluation framework.
- Challenge distribution: 51.0% of identified challenges involve Temporal Dynamics & Tracking, the largest share among the five categories.The remaining categories contain sufficient samples for comprehensive evaluation.
- Video duration: Video-Oasis spans clips under 15 seconds (19.5%) through videos exceeding 10 minutes (18.9%).This distribution covers both short and long videos.
- Answer distribution: Answer choices are reasonably balanced across A–D overall, although Global Narrative has 44.8% of answers outside A–D.Contributing benchmarks such as MMR-V use up to 12 answer choices.
- Cross-benchmark coverage: Cross-benchmark analysis finds complementary emphases: temporal reasoning in TVBench, spatial reasoning in VSI-Bench, and narrative reasoning in Video-Holmes and MMR-V.Video-Oasis aggregates these complementary challenges into one evaluation framework.
- Fine-Grained Perception: Fine-Grained Perception requires combining fragmented visual evidence across space and time, including partially visible objects and cluttered colors.The examples illustrate counting sofas across viewpoints and distinguishing non-white backgrounds.
- Spatial World Understanding: Spatial World Understanding integrates multi-view evidence to infer relative position, geometry, motion, and navigation.Examples require linking spatial relations across objects or using orientation and landmarks.
- Temporal Dynamics & Tracking: Temporal Dynamics & Tracking reconstructs ordered trajectories and action sequences rather than matching isolated frames.The examples include tracking a fish across positions and identifying the action following a bike ride.
- Causality & Logical Reasoning; Global Narrative: Causality & Logical Reasoning infers hidden causes or intentions, while Global Narrative connects dispersed events and evolving character states across the timeline.Examples include linking tickling to sneezing, attributing a crash to driver operation, identifying the snow leopard across ages, and inferring emotional change.
E Reproduction Results
The reproduction evaluation fixes a common frame-sampling protocol and compares reproduced results with official benchmark scores. The supplied passages state that the reproduced results remain wi.
- Evaluation protocol: Video-LLM reproduction uses a maximum of 128 frames at 1 fps for consistency across models.This differs from official settings that typically use 512 or 2048 frames.
- Reproduction comparison: The reproduced results remain wi.The supplied passage truncates the reported comparison, so no stronger result claim is supported.