Source-linked AI summary
MT-Video-Bench: A Holistic Video Understanding Benchmark for Evaluating Multimodal LLMs in Multi-Turn Dialogues
Yaning Pan, Qianqian Xie, Guohui Zhang, Zekun Wang, Yongqian Wen, Yuanxing Zhang, Haoxuan Hu, Zhiyu Pan, Yibing Huang, Zhidong Gan, Yonghong Lin, An Ping, Shihao Li, Yanghai Wang, Tianhao Peng, Jiaheng Liu
TL;DR
Existing video benchmarks primarily evaluate single-turn factual perception, leaving multi-turn video-grounded dialogue underassessed. MT-Video-Bench evaluates six perceptual and interactive capabilities across 1,000 curated dialogues. Evaluations reveal substantial model performance gaps, alongside declines associated with complex scenes and longer dialogue.
Problem
Existing video benchmarks predominantly focus on single-turn factual perception and do not provide a holistic framework for evaluating realistic multi-turn video-grounded dialogue.
Source: §1 · p. 1Method
MT-Video-Bench evaluates six perceptual and interactive capabilities across 1,000 curated multi-turn dialogues from varied video domains.
Source: §1 · p. 1Results
Gemini-2.5-Pro achieves 76.95% overall accuracy, while most open-source MLLMs score below 60% except the Qwen3-VL series, and performance declines with greater scene complexity and accumulating dialogue turns.
Source: §1 · p. 2Takeaways & Limitations
The benchmark offers a basis for studying models that reason over long contexts while engaging in natural, adaptive conversations.
Source: §5 · p. 8
Abstract
from arXiv · showhide
The recent development of Multimodal Large Language Models (MLLMs) has significantly advanced AI's ability to understand visual modalities. However, existing evaluation benchmarks remain limited to single-turn question answering, overlooking the complexity of multi-turn dialogues in real-world scenarios. To bridge this gap, we introduce MT-Video-Bench, a holistic video understanding benchmark for evaluating MLLMs in multi-turn dialogues. Specifically, our MT-Video-Bench mainly assesses 6 core competencies that focus on perceptivity and interactivity, encompassing 1,000 meticulously curated multi-turn dialogues from diverse domains. These capabilities are rigorously aligned with real-world applications, such as interactive sports analysis and multi-turn video-based intelligent tutoring. With MT-Video-Bench, we extensively evaluate various state-of-the-art open-source and closed-source MLLMs, revealing their significant performance discrepancies and limitations in handling multi-turn video dialogues. The benchmark will be publicly available to foster future research.
1 Introduction
Existing video benchmarks largely assess single-turn factual perception, leaving multi-turn video-grounded dialogue without a holistic evaluation framework. MT-Video-Bench evaluates six perceptual and interactive capabilities across 1,000 dialogues, and its results expose substantial performance gaps and challenges with scene complexity and extended dialogue.
1 Introduction
- Real-world interaction involves users refining questions and shifting topics, requiring models to integrate dialogue history and adapt while grounding responses in video.Existing benchmarks predominantly emphasize single-turn factual perception and neglect dialogue-level reasoning and the interplay of perceptivity and interactivity.Source: §1 · p. 1
- MT-Video-Bench assesses object reference, memory recall, content summary, answer refusal, topic shifting, and proactive interaction across 1,000 dialogues in varied video domains.The benchmark emphasizes cross-scene reasoning, long-range dependencies, and interactive adaptability.Source: §1 · p. 1
- Gemini-2.5-Pro reaches 76.95% overall accuracy, while most open-source MLLMs score below 60%, except the Qwen3-VL series.The authors characterize multi-turn perceptual and interactive capabilities as having significant room for improvement.Source: §1 · p. 2
- As scenes increase from 1–5 to over 20, Gemini-2.5-Pro’s fine-grained perception accuracy declines by 13%.The authors connect this pattern to long-range temporal dependencies and precise spatio-temporal reasoning in complex videos.Source: §1 · p. 2
- Performance declines as dialogue turns accumulate even with idealized golden context, while self-predicted context reveals an additional cascade effect from early errors.The paper identifies context reliability as increasingly critical in extended interactions.Source: §1 · p. 2
- The paper contributes a benchmark spanning 1,000 dialogues and 5,887 QA pairs, alongside experiments intended to identify challenges and potential improvement directions.The authors describe these findings as a roadmap for future research and development.Source: §1 · p. 2
2 Related Work
Related work covers advances in multimodal models for video and benchmarks that assess video understanding across different formats and temporal scales. The authors identify multi-turn dialogue, QA complexity and diversity, and comprehensive interactive assessment as remaining gaps.
Multimodal LLMs
- Recent multimodal LLM advances extend visual-language modeling to video understanding and grounded dialogue.The discussion names Qwen3-VL and InternVL3.5 as examples of recent systems.Source: §2 · pp. 2–3
Video Benchmarks
- Video benchmarks span multi-choice and open-ended QA, knowledge-intensive reasoning, long-video understanding, and real-time streaming dialogue.The comparison table describes benchmark dimensions including video length, QA counts and lengths, lexical diversity, and six capability labels.Source: §2 · p. 3, Table 1 · p. 3
- The authors identify gaps in multi-turn dialogue, QA complexity and diversity, and comprehensive interactive assessment across existing video benchmarks.These gaps motivate MT-Video-Bench.Source: §2 · p. 3
3 MT-Video-Bench
MT-Video-Bench evaluates perceptivity and interactivity in multi-turn video-grounded dialogue through six task capabilities. Its construction combines automated video and dialogue processing with human verification, and its evaluation uses curated dialogue history as golden context and checklist-based scoring.
3.1 Task Definition
- The benchmark is designed to evaluate perceptivity and interactivity, emphasizing contextual coherence, cross-scene comprehension, and adaptive interaction.Its six capabilities are object reference, memory recall, content summary, answer refusal, topic shifting, and proactive interaction.Source: §3.1 · p. 3
- Perceptivity covers resolving implicit references, retrieving prior dialogue information, and summarizing video and conversation while preserving semantic fidelity.Interactivity covers refusing unanswerable queries, adapting to topic changes, and sustaining engagement through clarifications or novel insights.Source: §3.1 · pp. 3–4
3.2 Dataset Construction
- Construction starts from manually collected and filtered videos, which are split into clips, captioned, and merged twice into coherent single-scene videos.Cross-scene processing samples frames at 2 FPS, filters for sharpness and redundancy, builds an object memory bank, and retrieves related scenes for merging.Source: §3.2 · p. 4
- Gemini-2.5-Pro generates dialogues for tasks scored 5 or 6 on a 1–6 scene-capability scale, with object-centered questions used for cross-scene dialogues.Two annotators independently review each dialogue for capability alignment, factual grounding, ambiguity, and coherence; disagreements go to a senior annotator for consensus.Source: §3.2 · pp. 4–5
3.3 Dataset statistics
- The 1,000 dialogues cover five categories and a relatively balanced distribution across six tasks, with videos of varied lengths and dialogues typically spanning 5–8 turns.The average dialogue contains 5.89 turns; most videos are under 15 minutes, while a small proportion exceed 15 minutes.Source: §3.3 · p. 5
3.4 Evaluation Method
- Evaluation supplies curated dialogue history as golden context rather than relying on potentially inconsistent model-predicted context.Gemini-2.5-Flash creates five yes/no checklist questions per QA pair, which are manually validated before automated response evaluation.Source: §3.4 · p. 5
- A dialogue score is checklist accuracy, and task performance averages dialogue scores within each task.The evaluation results table reports performance for object reference, memory recall, content summary, answer refusal, topic shifting, and proactive interaction.Source: §3.4 · p. 6, Table 2 · p. 6
4 Experiments
The experiments compare open- and closed-source MLLMs on MT-Video-Bench and analyze how task, dialogue, scene, input, and video-length conditions affect performance. Gemini-2.5-Pro leads the reported comparisons, while the analyses identify degradation under more demanding conditions and high agreement between automated and human evaluation.
4.1 Experimental Settings
- The evaluation includes three closed-source models and 21 representative open-source MLLMs.Most models use 128 uniformly sampled frames; image resizing follows model-specific settings.Source: §4.1 · p. 6
4.2 Main Results
- Gemini-2.5-Pro leads across evaluation metrics, while Qwen3-VL-32B-Thinking surpasses Doubao-Seed-1.6 in overall accuracy.CS is the most mastered task, with top models frequently exceeding 90% success, while AR remains a bottleneck; reasoning mode improves results across model scales.Source: §4.2 · p. 7
4.3 Further Analysis
- As scene counts increase, Gemini-2.5-Pro declines, with MR and TS degrading most; golden context outperforms self-predicted or absent context for Qwen3-VL-8B-Instruct.The cited analysis describes a substantial gap between golden and self-predicted context, indicating error propagation.Source: §4.3 · p. 7, Figure 4 · p. 7
- Performance decays as dialogue turns increase, and video length beyond 15 minutes brings a significant drop across models.InternVL3.5-8B (No Think) falls from 57.18 to 34.63 between 5 and 8 turns; most models peak at 720p, while frame-count gains can saturate for smaller models.Source: §4.3 · pp. 7–8, Figure 5 · p. 8
4.4 Case Study
- The error analysis identifies visual-temporal hallucinations, inter-turn interference, and conversational passivity as three primary failure modes.Examples of these failures are provided in Appendix C.Source: §4.4 · p. 8
4.5 Agreement Evaluation
- Automated evaluation shows high agreement with human judgments across the reported correlation metrics, with Gemini-2.5-Flash achieving the highest consistency.The study sampled 20 dialogues per task, totaling 120, and used three human annotators.Source: §4.5 · p. 8
5 Conclusion
MT-Video-Bench evaluates multi-turn video dialogue through six capabilities spanning perceptivity and interactivity. The authors report evaluations of 20 state-of-the-art models and position the benchmark as a foundation for research on long-context reasoning and adaptive conversation.
5 Conclusion
- MT-Video-Bench jointly assesses perceptivity and interactivity through six capabilities, including memory recall, topic shifting, and proactive interaction.The paper contrasts this scope with prior benchmarks that primarily focus on single-turn factual perception.Source: §5 · p. 8
- The authors evaluate 20 state-of-the-art models and present the benchmark as a foundation for future research.The conclusion highlights the need for models that reason over long contexts while engaging in natural, adaptive conversations.Source: §5 · p. 8
- The benchmark's stated research focus is models that can reason over long contexts while engaging in natural, adaptive conversations.Source: §5 · p. 8
A.1 Distribution of Video Category
The appendix describes the benchmark's video-category coverage and its structured framework for generating multi-turn dialogues.
A.1 Distribution of Video Category
- The video collection covers English and Chinese content published within the past year, with categories spanning a diverse range of domains.Figure 7 illustrates the distribution of video domains.Source: §A.1 · p. 12, Figure 7 · p. 12
A.2 Details on generation prompts
- Dialogue generation uses a unified prompt template for conversational persona and global context, supplemented by task-specific instructions for six evaluation tasks.The instructions target contextual coherence, fine-grained spatio-temporal reasoning, and adaptive interaction.Source: §A.2 · p. 12
B Details on Evaluation
The evaluation appendix outlines inference settings and the prompts used for checklist generation and final evaluation. It states that inference uses a unified template and follows model developers' recommendations for model-specific settings.
B Details on Evaluation
- The evaluation setup covers inference settings, checklist-generation prompts, and the final evaluation process.Source: B Details on Evaluation · p. 12
B.1 Inference Settings
- A unified inference prompt template is used to standardize evaluation across diverse MLLM architectures.Source: §B.1 · p. 12
- InternVL3.5 Think mode follows its official system-prompt protocol, and inference hyperparameters follow each model developer's recommendations.The listed hyperparameters include temperature, top-p, and top-k.Source: §B.1 · p. 12
B.2 Prompt for Generating Evaluation Checklists
The checklist-generation prompt instructs an evaluator to create a verification checklist for comparing a model’s answer with the ground truth.
B.2 Prompt for Generating Evaluation Checklists
- Figure 16 presents the instructions used to generate comprehensive checklists for assessing model performance.Source: §B.2 · p. 12
B.3 Prompt for Evaluation Based on Checklists
The evaluation prompt applies generated checklists to assess model responses.
B.3 Prompt for Evaluation Based on Checklists
- The evaluation prompt uses generated checklists to evaluate model responses.Source: §B.3 · p. 12
C More Cases
This appendix provides prompt and dialogue examples for MT-Video-Bench, including task instructions, checklist generation, and cases illustrating model responses.
C More Cases
- Red text marks erroneous response segments in the cases, and square brackets in checklists enclose correct answers.Source: C More Cases · p. 12
- The topic-shifting instructions cover user moves to video-related or unrelated topics and call for an appropriate response.Source: C More Cases · p. 13
- Answer-refusal instructions require declining questions about video content that is absent, while grounding answers in video evidence or logical inferences.Source: C More Cases · p. 14
- The checklist-generation prompt takes a user question and ground-truth answer as inputs for an expert evaluator creating a verification checklist.The accompanying evaluation process uses generated checklists to judge model responses.Source: C More Cases · p. 17
- The examples include multi-turn dialogues about sugarcane juice, a HUNTRX fan-event rivalry, an airplane scene, and paper crafts.Source: C More Cases · pp. 18, 20, 21, 23
- In the answer-refusal case, the assistant says the clip shows no added fruit or ingredients and limits its description to sugarcane pressing and juice collection.Source: C More Cases · p. 18