Source-linked AI summary
Video-MME-Logical: A Controlled Diagnostic Benchmark for Video Temporal-Logical Reasoning
Hohin Kwan, Hongyu Li, Ray Zhang, Manyuan Zhang, Xianghao Kong, Anyi Rao, Jiahao Xie, Si Liu
TL;DR
MLLMs may aggregate frames without performing robust temporal-logical reasoning, while existing benchmarks do not isolate or diagnostically measure this capability. Video-MME-Logical addresses this gap with controlled, difficulty-scaled tasks and intermediate-state diagnostics, revealing a substantial human-model gap and limited gains from supervised scaling.
Problem
Existing video benchmarks do not isolate temporal-logical reasoning from general temporal understanding, making model errors and capabilities difficult to interpret.
Method
Video-MME-Logical organizes 25 procedurally generated tasks around five temporal-logical operations with controlled difficulty and verifiable intermediate-state diagnostics.
Results
40% accuracy at 375K samples was achieved through supervised fine-tuning, but further scaling provided no clear gains and a substantial human-model gap remained.
Takeaways & Limitations
Video-MME-Logical offers a scalable diagnostic testbed for analyzing video temporal-logical reasoning and reasoning failures in MLLMs.
Takeaways & Limitations
Procedurally generated videos differ from natural videos in visual appearance, scene diversity, and real-world ambiguity.
Abstract
from arXiv · showhide
Recent interest in multimodal large language models (MLLMs) raises a central question: can they reason over dynamic visual evidence rather than merely recognize objects or events in individual frames? This ability, which we refer to as video temporal-logical reasoning, requires models to maintain, update, and compose evidence as visual states evolve across frames. Existing video benchmarks often conflate this capability with scene complexity, static recognition, or uncontrolled temporal variation. To isolate this capability, we introduce Video-MME-Logical, a controlled benchmark organized around five temporal-logical operations: state tracking, sequential counting, temporal ordering, dynamic spatiality, and structural composition. The benchmark contains 25 fine-grained task categories generated with controlled object states, transitions, temporal dependencies, and logical compositions. It enables difficulty-controlled final-answer evaluation by varying temporal horizon and reasoning complexity, and supports intermediate-state diagnostics by verifying whether models recover the required logical reasoning trace before producing the final answer. Experiments with state-of-the-art MLLMs reveal a substantial human-model gap, especially as temporal-logical complexity increases. Supervised fine-tuning on up to 500K generated samples improves performance but remains insufficient to close the reasoning gap, positioning Video-MME-Logical as a scalable testbed for analyzing and improving temporal-logical reasoning in MLLMs.
1 Introduction
Video-MME-Logical addresses the gap between multi-frame aggregation and temporal-logical reasoning with a controlled benchmark organized around five operations and 25 fine-grained tasks. Its difficulty-controlled and intermediate-state evaluations expose a substantial human-model gap, while scaling supervised fine-tuning improves performance but remains insufficient to close it.
- Introduction: Existing video benchmarks often organize evaluations by data source, scene type, event category, or action class rather than explicit temporal-logic operations.Consequently, strong multi-frame performance does not establish logical reasoning over time.
- Introduction: VIDEO-MME-LOGICAL organizes temporal-logical reasoning into five operations and instantiates them as 25 fine-grained task categories.The operations are State Tracking, Sequential Counting, Temporal Ordering, Dynamic Spatiality, and Structural Composition.
- Introduction: Procedural generation controls object states, state transitions, temporal dependencies, and logical compositions across three difficulty levels defined by temporal horizon and reasoning complexity.This design targets the lack of controlled diagnostic settings that isolate temporal-logical reasoning from general temporal understanding.
- Introduction: Difficulty-controlled evaluation and intermediate-state diagnostics verify reasoning evidence beyond final answers and reveal a substantial human-model gap.The VIDEO-MME-LOGICAL-S subset supports describing and verifying intermediate evidence before producing the final answer.
- Introduction: 40% accuracy at 375K samples shows that supervised fine-tuning improves Qwen3-VL-8B, but further scaling yields no clear gains and remains insufficient to close the human-model gap.The study generates 500K procedurally created training samples and fine-tunes Qwen3-VL-8B at different data scales.
2 Related Work
Existing video benchmarks span broad comprehension and targeted temporal or spatiotemporal reasoning, but they leave temporal-logical operations and difficulty-controlled reasoning traces under-specified. Video-MME-Logical addresses this gap with synthetic, controllable videos designed to isolate temporal-logical reasoning across multiple dimensions.
- General Video Understanding Benchmarks: General video benchmarks expand evaluation across domains, durations, task formats, and open-ended instruction following, while covering egocentric activities, temporal relations, object interactions, and perception.Examples include Video-MME (Fu et al., 2025), MLVU (Zhou et al., 2025), ALLVB (Tan et al., 2025), LongVideoBench, LVBench (Wang et al., 2025a), CinePile (Rawal et al., 2024), VideoEspresso, MovieChat (Song et al., 2024), Ego4D (Grauman et al., 2022), EgoSchema (Mangalam et al., 2023), MVBench (Li et al., 2024a), NExT-QA, and the Perception Test (Patraucean et al., 2023).
- General Video Understanding Benchmarks: Their intentional heterogeneity rarely organizes questions around fixed logical categories or difficulty-controlled traces, leaving well-defined temporal-logical operations under-specified.VIDEO-MME-LOGICAL instead uses synthetic, controllable videos to isolate temporal-logical reasoning across multiple dimensions.
- Video Temporal Reasoning Benchmarks: Targeted benchmarks probe fine-grained temporal concepts, ordered observations, cross-segment reasoning, what-when-where chains, and complex vision-centric video reasoning.VITATECS (Li et al., 2024b), ReXTime (Chen et al., 2024), V-STaR (Cheng et al., 2025), and VideoReasonBench (Liu et al., 2026) represent these directions.
- Video Temporal Reasoning Benchmarks: These temporal-reasoning benchmarks show that current models struggle with temporal cues, event ordering, cross-time relations, and spatiotemporal grounding.The reported difficulties span both temporal structure and the grounding of information across space and time.
- Video Temporal-Logical Reasoning: Recent temporal-logical studies expose failures in temporal tracking and video reasoning but do not systematically distinguish the temporal-logical operation required by an MLLM.Liu and Lee (2026) provides a controlled but task-specific shell-game-style tracking test, while Wang et al. (2026) emphasizes broad video-reasoning behaviors at scale.
3 VIDEO-MME-LOGICAL Benchmark
VIDEO-MME-LOGICAL is a programmatically generated benchmark that isolates temporal-logical reasoning through five operations, 25 controlled task categories, and difficulty-scaled evaluation. Its structured metadata supports exact answer computation and intermediate-state process verification.
- Task Architecture: The benchmark organizes temporal-logical reasoning into state tracking, sequential counting, temporal ordering, dynamic spatiality, and structural composition.These abilities require models to remember information, update it as visual states change, and compose multiple temporal states for inference.
- Dataset Statistics: VIDEO-MME-LOGICAL contains 503,750 videos, comprising 500K training videos and 3,750 test videos across 25 categories and three difficulty levels.The difficulty levels—easy, medium, and hard—vary by category-specific temporal horizons and reasoning complexity factors; an 8-category subset supports intermediate-state diagnostics.
- Data Curation Pipeline: Programmatic generation decouples temporal-logical reasoning from visual noise and annotation ambiguity while enabling reproducible generation, controllable difficulty, and verifiable intermediate evidence.The approach is inspired by synthetic benchmarks including CLEVR (Johnson et al., 2017; Yi et al., 2020; Zhuo et al., 2025).
- Data Curation Pipeline: Each task program combines temporal transitions, scene configuration, metadata construction, and reasoning components to control video evolution, visual content, evidence traces, and inference.Metadata records temporal evidence for rendering, question generation, answer computation, difficulty control, and intermediate-state supervision.
- Data Curation Pipeline: Question-answer pairs are generated from program metadata with exact automatic answers, while training traces and intermediate states support reasoning supervision and process verification.Evaluation counts answers correct only when predicted options, tagged answers, or canonicalized intermediate structures exactly match the ground truth; overall accuracy is Acc = Ncorrect/Ntotal.
4 Experiments
Experiments evaluate final answers, intermediate reasoning states, and supervised fine-tuning scaling. Current MLLMs remain far below humans, while intermediate-state diagnostics and data scaling reveal limitations in reasoning reliability and generalization.
- Final-answer evaluation: Human performance reaches 95.9% overall accuracy, whereas the strongest evaluated model, gemini-3.1 Pro, achieves only 28.6% on VIDEO-MME-LOGICAL.Table 2 reports final-answer accuracy across all 25 task categories and three difficulty levels.
- Final-answer evaluation: gemini-3.1 Pro drops from 33.1% to 20.6% on harder settings, a 12.5% degradation that remains far below human performance.The smaller degradation suggests stronger robustness, but longer temporal horizons and higher reasoning complexity remain challenging.
- Intermediate-state evaluation: Intermediate-state evaluation exposes failures hidden by final answers: GPT-5.4 reaches 63.0% on Count.-S versus gemini-3.1 Pro’s 35.0% in easy settings.GPT-5.4 also leads on Order.-S, at 27.0% versus 13.0%, while gemini-3.1 Pro leads several corresponding final-answer categories.
- Supervised fine-tuning scaling: Increasing SFT data to 375K raises overall accuracy to 39.2%, but 500K reduces it to 37.7%, indicating clear yet saturating gains.Training uses Qwen3-VL-8B with balanced samples from five data sizes; easy-level training is evaluated on medium and hard settings to test generalization.
- Supervised fine-tuning scaling: Ours-375K-Thinking reaches 54.8% Avg. E, showing transfer from easy-level training, while harder-setting generalization remains limited.Verifiable intermediate states can distinguish genuine temporal-logical reasoning from superficially correct final answers with flawed intermediate reasoning.
5 Conclusion
VIDEO-MME-LOGICAL is a controlled diagnostic benchmark for video temporal-logical reasoning that organizes 25 tasks around five operations and uses difficulty-controlled settings with intermediate-state diagnostics. Experiments reveal a substantial human-model gap, especially with longer temporal horizons, higher reasoning complexity, and process-level evaluation.
- VIDEO-MME-LOGICAL introduces a controlled diagnostic benchmark for video temporal-logical reasoning.
- The benchmark organizes 25 tasks around five temporal-logical operations to test whether MLLMs maintain, update, and compose visual evidence over time.
- Difficulty-controlled settings and intermediate-state diagnostics isolate temporal-logical reasoning across final answers and reasoning processes.
- Experiments reveal a substantial human-model gap, especially under longer temporal horizons, higher reasoning complexity, and process-level evaluation.
Limitations
The benchmark’s procedural video generation enables scalable, controllable, and verifiable evaluation but creates a gap from natural videos in appearance, diversity, and ambiguity. Large-scale natural-video annotation remains difficult because reliable temporal states, exact answers, and process-level supervision are hard to obtain.
- Limitations: Procedurally generated videos provide scalable data generation, controllable difficulty, and verifiable intermediate states, but differ from natural videos in visual appearance, scene diversity, and real-world ambiguity.This gap motivates viewing the benchmark as a controlled diagnostic setting rather than a direct substitute for natural-video evaluation.
- Limitations: Large-scale annotation of natural videos is difficult because reliable temporal states, exact answers, and process-level supervision are hard to obtain.
A Benchmark Details and Task Taxonomy
The benchmark details section organizes task categories by cognitive category and specifies whether answers use multiple-choice or open-ended formats. A word cloud provides an additional overview of VIDEO-MME-LOGICAL.
- Task Taxonomy: Task categories are grouped by cognitive category, with answers formatted as multiple-choice or open-ended numeric, string, or JSON responses.Table 5 defines MC and Fill-in formats for the benchmark’s task categories.
- Benchmark Overview: A word cloud offers a visual overview of VIDEO-MME-LOGICAL.
B Training Configuration for the Qwen3-VL-8B SFT Experiments
This section presents the training configuration used for the Qwen3-VL-8B supervised fine-tuning experiments.
- The training configuration for the Qwen3-VL-8B SFT experiments is summarized in Table 6.
C Human Evaluation Protocol
The human evaluation estimated human-level performance using 3,750 sampled examples answered by three annotators under the same visual-input conditions as model evaluation.
- C Human Evaluation Protocol: Human performance was estimated from 3,750 VIDEO-MME-LOGICAL examples evaluated by three annotators.Annotators saw each video and corresponding question but not program metadata, ground-truth answers, intermediate-state annotations, or model predictions.
D More Visualization Analysis
This section presents additional visual examples from VIDEO-MME-LOGICAL across Figures 6–12.
- Figures 6–8 show additional visual examples from VIDEO-MME-LOGICAL.
- Figures 9–10 show additional visual examples from VIDEO-MME-LOGICAL.
- Figures 11–12 show additional visual examples from VIDEO-MME-LOGICAL.