Source-linked AI summary
OVIBench: Benchmarking Online Video Question Answering under Interruption
Naiming Liu, Zhiheng Wu, Shuning Wang, Tie Zhang, Bowen Liu, Tong Wang
TL;DR
Existing video QA benchmarks mainly use offline, single-round interactions, while real users may interrupt generation. OVIBench formalizes and evaluates interruption handling, and OVI-Train fine-tuning substantially improves performance.
Problem
Existing video QA research largely evaluates offline, single-round answering rather than interactions where users interrupt generation and require intent-aware response adjustment.
Method
The paper introduces OVIBench with three interruption types, open-ended and multiple-choice formats, offline simulation, six metrics, and OVI-Train for interruption-aware fine-tuning.
Results
Interruption-aware fine-tuning improves multiple-choice accuracy by 16.88% and interruption-type accuracy by 13.94%, with the fine-tuned 7B model surpassing larger models in reported comparisons.
Takeaways & Limitations
OVIBench exposes persistent interruption-handling weaknesses, while targeted training significantly improves responsiveness in online video question answering.
Takeaways & Limitations
The offline protocol cannot fully capture real systems' generation speed, network latency, and user input timing, and the benchmark covers only three interruption types.
Abstract
from arXiv · showhide
Recent vision language models (VLMs) have achieved strong progress in video understanding. However, most existing video QA research and benchmarks still follow an offline, single-round paradigm, overlooking realistic interactions where users may interrupt the model during answer generation. To address this gap, we formulate the task of Online Video Question Answering under Interruption and introduce OVIBench, the first standardized benchmark for evaluating VLMs in this setting. OVIBench categorizes interruptions into three types: Cancellation, False Trigger, Correction and supports both open-ended and multiple-choice evaluations. To enable large-scale and reproducible testing, we develop an offline simulation protocol that reproduces interruption during generation under a unified temporal setup, together with a multi-dimensional metric suite for assessing interruption understanding and response generation. Experiments demonstrate that OVIBench effectively distinguishes models' interruption-handling abilities, especially in following correction requests. Finally, we construct a train set OVI-Train for interruption-aware fine-tuning. Models fine-tuned on this dataset achieve significant gains on OVIBench, validating the effectiveness of our benchmark and data design. OVIBench, OVI-Train, and the evaluation code will be released.
1 Introduction
OVIBench addresses the mismatch between offline single-round video QA and interactive settings where users interrupt generation. It formalizes interruption types, supports reproducible evaluation, measures interruption handling, and introduces interruption-aware training data.
- Existing video QA benchmarks typically evaluate complete videos and questions in a single pass, unlike interactions with mid-generation user instructions.
- OVIBench formalizes online video QA interruptions as Cancellation, False Trigger, and Correction, with open-ended and multiple-choice evaluation.
- Its offline simulation protocol reproduces interruption during generation under a unified temporal setup for large-scale, reproducible evaluation.
- The metric suite evaluates interruption-intent recognition, intent fulfillment, fluency, content consistency, and consistency with observed video evidence.
- Mainstream MLLMs remain weak under online interruptions, particularly when following correction requests.
- OVI-Train supports interruption-aware fine-tuning; its 7B model improves multiple-choice accuracy by 16.88% and interruption-type accuracy by 13.94% over Qwen2.5-VL-7B.
2 Related Work
Related work has advanced offline video understanding and interactive dialogue evaluation, but the cited research does not establish a benchmark for interruption handling in online video QA.
- Video-LLMs and Online Video Understanding: Recent MLLMs improve offline video understanding through stronger visual perception modules and staged training strategies.
- Interactive Modeling and Turn-Taking Benchmarks: Speech-dialogue research treats turn-taking as central to natural interaction and benchmarks pause processing, backchannel behavior, and interruption-related conversational events.
3 OVIBench
OVIBench defines interruption-aware online video QA around partial video, questions, generated answers, and interruption signals. Its benchmark combines structured interruption taxonomy, generated open-ended and multiple-choice data, and offline temporal simulation.
- 3.1 Task Formulation: At interruption time, the model observes video frames, the initial question, its partial answer, and the interruption signal before producing a post-interruption response.
- 3.2 Interruption Taxonomy: The task classifies interruptions as False Trigger, Correction, or Cancellation and couples intent identification with continuing, revising, or terminating generation.
- 3.3 Data Collection: The dataset construction uses 3,200 videos from five public datasets covering open-domain activities, narrative dialogue, highlights, anomalous events, and instructional procedures.
- 3.4 QA Generation and Data format: VLM-generated questions are designed to elicit sufficiently long raw answers, after which interruption type, signal, and randomly sampled interruption time are inserted.
- 3.4 QA Generation and Data format: The multiple-choice format pairs four options with a ground-truth interruption-type label: three type-specific correct answers and one VLM-generated distractor.
- 3.5 Simulate Online Video Interruption: Offline simulation fixes answer-generation rate, truncates the response at the interruption time, reconstructs observed video, and evaluates the model's new response to the signal.
4 Model-Judged Evaluation of OVIBench
OVIBench evaluates interruption handling with model-judged open-ended responses across intent recognition, fulfillment, fluency, evidence consistency, and correction behavior. Existing models remain weak on corrections, although performance varies by metric and model.
- OVIBench compares state-of-the-art VLMs using interruption classification and complementary response-quality metrics in open-ended evaluation.The metrics cover interruption understanding, intent fulfillment, textual fluency, visual evidence, and content consistency.
- Most models score low on CFS and CCS, revealing persistent difficulty executing user corrections and maintaining content consistency.
- Doubao-Seed-1.6 achieves the best overall results, ranking highest in DAC, TFS, CFS, CCS, and overall accuracy.
- Qwen3-VL-8B leads VES and IFA, indicating strong video–response consistency and effective interruption-intent fulfillment.
- Cancellation accuracy approaches 100%, whereas False Trigger varies most across models and Correction typically reaches 87%–92%.False Trigger is especially difficult for smaller models; Correction is intermediate in difficulty.
5 Multiple-Choice Evaluation of OVIBench
The multiple-choice evaluation combines OVI-Train construction with low-noise measurement of answer selection and interruption-type recognition. Fine-tuning substantially improves a Qwen2.5-VL-7B model, including gains over larger and specialized comparators.
- OVI-Train contains 20765 training instances designed to provide explicit interruption-type supervision for interruption-aware fine-tuning.Samples pair the question, truncated pre-interruption answer, interruption instruction, video segment, candidate options, and supervision label.
- The OVI-Train procedure uses truncated answers and injected interruption instructions to train models to select the correct interruption type.
- Multiple-choice evaluation reports MA for answer selection and CA for interruption-type classification, reducing subjectivity while measuring intent recognition.
- After fine-tuning, the 7B model reaches 82.42% MA and 95.33% CA on the full set, compared with 65.54% and 81.39% for the baseline.
- The largest gains are False Trigger MA (+52.94%) and Cancellation CA (+49.08%), and the fine-tuned 7B model surpasses Qwen2.5-VL-72B on several metrics.
6 Conclusion
The paper introduces OVIBench for online video QA under interruption, using offline simulation and multidimensional evaluation to measure realistic interruption handling. OVI-Train fine-tuning improves responsiveness, providing a framework for evaluation and improvement.
- OVIBench formalizes online video QA under interruption with three interruption types and both open-ended and multiple-choice evaluation.
- Offline interruption simulation enables large-scale, reproducible evaluation, while six complementary metrics assess model performance.
- Existing MLLMs struggle under interruptions, motivating OVI-Train for interruption-aware fine-tuning.
- Specialized training significantly improves model responsiveness in online video question answering.
7 Limitations
OVIBench’s evaluation framework has boundaries stemming from its offline simulation, limited interruption taxonomy, and VLM-generated data. The authors identify broader interactive settings, richer interruption scenarios, and more human data as directions for extension.
- Offline interruption simulation may not capture dynamic generation speed, network latency, or user input timing in real online systems.
- The benchmark covers Cancellation, False Trigger, and Correction, leaving multi-turn follow-ups, task switching, and mixed intentions for future exploration.
- VLM-generated questions, interruption signals, and candidate answers support scalable construction, but additional human annotations or real user data could improve naturalness and coverage.
B Data Format
OVIBench represents interrupted video-QA interactions in open-ended and multiple-choice formats. The formats preserve temporal interaction state while trading off realistic free-form generation against controlled, reproducible response selection.
- B Data Format: Each benchmark instance is organized around a temporally grounded interrupted interaction with complementary open-ended and multiple-choice formats.
- B.1 Open-Ended Format: Open-ended samples contain the original question, question timestamp, interruption signal, interruption timestamp, and source video.
- B.1 Open-Ended Format: Each open-ended record stores a unique identifier, interruption type, and timestamped interaction setup.
- B.1 Open-Ended Format: During open-ended evaluation, the model answers the original question before responding to the interruption under the specified temporal setup.
- B.2 Multiple-Choice Format: The multiple-choice format reduces evaluation noise by asking models to select the candidate answer that best matches the interruption context.
- B.2 Multiple-Choice Format: Multiple-choice distractors are designed to resemble plausible responses while being semantically incorrect rather than randomly irrelevant.
- B.2 Multiple-Choice Format: Multiple-choice samples record dialogue history, question, partial answer, interruption signal and timestamp, visual range, four options, and a ground-truth label, with shuffled option order.
- B.2 Multiple-Choice Format: For a Cancellation signal such as “That’s enough,” the correct response is [STOP], illustrating controlled interruption understanding and response selection.
C Full Experimental Results
Open-ended results show that interruption handling depends strongly on interruption type. Cancellation is relatively easy, Correction remains difficult, and False Trigger varies most across models, with six metrics illustrated for evaluation.
- Interruption handling is strongly type-dependent: Cancellation is relatively easy, Correction remains challenging, and False Trigger varies most across models.
- The open-ended results are reported separately for Cancellation, Correction, and False Trigger in Tables 12–14.
- Figure 5 illustrates six evaluation metrics through short video streams, interruptions, and representative responses, marking high scores in green and failures in red.
D Visualization of Evaluation Metrics
The visualization uses example interrupted video interactions to make OVIBench’s metric suite concrete. It contrasts successful and failed responses while showing the distinct aspects each metric evaluates.
- Figure 5 presents short video streams, user interruptions, and representative model responses for illustrating OVIBench’s six evaluation metrics.
- Green boxes mark high-quality responses, whereas red boxes identify typical failure cases.
- The metrics cover interruption intent, textual fluency, visual grounding, request fulfillment, correction following, and preservation of valid context.
E Human Evaluation Subset
OVI-Human is a held-out, human-written evaluation subset designed to test whether interruption-handling gains generalize beyond automatically generated data. It uses natural interruption expressions and evaluates model behavior with human-scored quality criteria and multiple-choice metrics.
- Evaluation protocol: OVI-Human was excluded from OVI-Train training, prompt tuning, and model selection, serving only as an additional evaluation set.This design tests whether conclusions hold under manually written questions and interruption expressions.
- Subset construction: The subset uses diverse questions, natural interruption expressions, and manually written or substantially rewritten answer options.Examples include implicit stop requests, colloquial corrections, irrelevant insertions, counting, localization, and causal reasoning.
- Results: 17.78 points in All MA and 15.00 points in All CA separated the fine-tuned model from Qwen2.5-VL-7B on OVI-Human.The strongest improvements were reported for False Trigger and Cancellation, and the overall trend matched the main multiple-choice evaluation.
- Human evaluation: Human evaluation assessed interruption-type correctness, response appropriateness, visual consistency, contextual coherence, and multiple-choice option quality.Cancellation should stop, False Trigger should continue the original answer, and Correction should revise the response while preserving unaffected context.
- Human evaluation: Human scores combined binary correctness judgments with 1–10 ratings averaged across 10 participants.The rated dimensions included textual fluency, visual consistency, correction quality, and contextual coherence.
G Offline Simulation Pseudocode
The offline simulation protocol reproduces interruption during answer generation by controlling observable video context, timing, answer truncation, and post-interruption querying. It records predicted interruption types and post-interruption responses for reproducible batch evaluation.
- Post-interruption reasoning: After interruption, the protocol updates the observable video context and constructs a prompt from the interruption signal and task instructions.The updated context and partial response are passed into the post-interruption query.
- Inputs: The simulation parses each benchmark item into video, question, interruption signal, question timestamp, and interruption timestamp.These fields define the inputs and temporal structure for the simulated interaction.
- Initial answering stage: The initial answering stage uses video observed before the question timestamp to generate a complete preliminary answer.The observable segment is VQ ← V[0 : T0], and the answer is generated from that segment and the question.
- Post-interruption reasoning: The model receives updated context, the original question, and the partial response, then outputs an interruption type and post-interruption response.The output Y is parsed into predicted type c-hat and response A-prime.
- Output recording: Each simulation record stores the question, interruption data, timestamps, full and partial answers, predicted type, and post-interruption response.The records are accumulated into a reproducible simulation result set.