PaperlayerView exact revision on arXiv

Source-linked AI summary

OVO-Bench: How Far is Your Video-LLMs from Real-World Online Video Understanding?

Yifei Li, Junbo Niu, Ziyang Miao, Chunjiang Ge, Yuanhang Zhou, Qihao He, Xiaoyi Dong, Haodong Duan, Shuangrui Ding, Rui Qian, Pan Zhang, Yuhang Zang, Yuhang Cao, Conghui He, Jiaqi Wang

arXiv:2501.05510v2cs.CVcs.AI

TL;DR

Existing benchmarks inadequately evaluate timestamp-dependent online video understanding. OVO-Bench introduces a curated, multi-mode benchmark and temporal evaluation pipeline, finding substantial shortcomings in current models while showing competitive offline-model performance for real-time perception.

  • Problem

    Existing evaluations inadequately measure timestamp-dependent online video understanding across past, current, and future information.

  • Method

    OVO-Bench combines 644 videos, 2814 human-curated timestamped meta-annotations, 12 tasks, and dense temporal querying across three online-understanding modes.

  • Results

    Current Video-LLMs struggle with online video understanding, while offline models perform competitively in Real-Time Visual Perception tasks.

  • Takeaways & Limitations

    OVO-Bench provides a resource for rigorous evaluation and future development of Video-LLMs toward practical online video understanding.

  • Takeaways & Limitations

    Dataset diversity may be constrained by scarce suitable annotations, unsatisfactory automatic QA generation, and high human annotation costs, introducing potential bias.

Abstract

from arXiv · show

Temporal Awareness, the ability to reason dynamically based on the timestamp when a question is raised, is the key distinction between offline and online video LLMs. Unlike offline models, which rely on complete videos for static, post hoc analysis, online models process video streams incrementally and dynamically adapt their responses based on the timestamp at which the question is posed. Despite its significance, temporal awareness has not been adequately evaluated in existing benchmarks. To fill this gap, we present OVO-Bench (Online-VideO-Benchmark), a novel video benchmark that emphasizes the importance of timestamps for advanced online video understanding capability benchmarking. OVO-Bench evaluates the ability of video LLMs to reason and respond to events occurring at specific timestamps under three distinct scenarios: (1) Backward tracing: trace back to past events to answer the question. (2) Real-time understanding: understand and respond to events as they unfold at the current timestamp. (3) Forward active responding: delay the response until sufficient future information becomes available to answer the question accurately. OVO-Bench comprises 12 tasks, featuring 644 unique videos and approximately human-curated 2,800 fine-grained meta-annotations with precise timestamps. We combine automated generation pipelines with human curation. With these high-quality samples, we further developed an evaluation pipeline to systematically query video LLMs along the video timeline. Evaluations of nine Video-LLMs reveal that, despite advancements on traditional benchmarks, current models struggle with online video understanding, showing a significant gap compared to human agents. We hope OVO-Bench will drive progress in video LLMs and inspire future research in online video reasoning. Our benchmark and code can be accessed at https://github.com/JoeLeelyf/OVO-Bench.

1. Introduction

Existing video benchmarks poorly represent real-world online understanding because they rely on coarse, short-video questions with limited temporal dependence. OVO-Bench addresses this gap by evaluating timestamp-sensitive reasoning across backward, real-time, and forward-response modes.

  • Motivation: Existing benchmarks often use coarse-grained QA on short videos with fixed question types and limited temporal indispensability.These evaluations leave a domain gap between benchmark performance and real-world video understanding.
  • Motivation: StreamingBench evaluates immediate responses using existing visual inputs, but its categories incompletely represent streaming perception.The benchmark does not fully test how models should respond as visual information unfolds over time.
  • Benchmark design: Effective online understanding requires tracing past information, perceiving current events, and actively responding when future evidence is needed.OVO-Bench frames this as a Video Chain-of-Time thinking process in which models decide whether to answer immediately or wait.
  • Benchmark design: OVO-Bench contains 644 videos, 2814 human-curated timestamped meta-annotations, and 12 tasks spanning three online-understanding categories.The categories are Backward Tracing, Real-Time Visual Perception, and Forward Active Responding, with videos covering seven domains and durations from minutes to half an hour.
  • Evaluation: The evaluation pipeline queries Video-LLMs densely along temporal axes and adapts offline models through temporally truncated video segments.This setup enables systematic testing of continuous information processing and explores whether offline models can support online understanding.
  • Findings: Streaming models show a wider performance gap than offline counterparts, while offline Video-LLMs perform competitively in real-time visual perception.Together, the reported findings indicate uneven transfer from offline video comprehension to online settings.

2. Real-Time Visual Perception:

Real-Time Visual Perception evaluates immediate understanding of ongoing visual content, including spatial, object, attribute, action, and text recognition. OVO-Bench builds these evaluations from timestamped annotations and curated or generated question-answer pairs across diverse videos.

  • Real-Time Visual Perception:: OVO-Bench organizes online perception into Backward Tracing, Real-Time Visual Perception, and Forward Active Responding.The benchmark taxonomy distinguishes retrieving past information, responding to current information, and waiting for future evidence.
  • Real-Time Visual Perception:: Real-Time Visual Perception tests immediate understanding of ongoing visual content at the current timestamp.The section frames real-time perception as essential when present actions shape future outcomes.
  • Real-Time Visual Perception:: Its core capabilities include spatial, object, attribute, action, and optical character recognition.These categories cover relationships between nearby-frame objects, current-frame objects and actions, object properties, and visible characters.
  • Real-Time Visual Perception:: The benchmark uses timestamped video data, with sources selected from existing datasets and self-crawled videos to increase diversity.The construction process addresses the scarcity of event-level timestamps through a metadata-generation pipeline.
  • Real-Time Visual Perception:: Questions combine existing QA pairs with automatically generated candidates and human-curated prompts tailored to real-time queries.Multiple-choice options are used for Backward Tracing and Real-Time Visual Perception, while prompts can include cues such as “Now” or “Currently.”
  • Real-Time Visual Perception:: The dataset contains 644 videos across seven domains and 2,814 QA pairs, with multiple-choice questions offering two to five options.Video durations range from a few minutes to half an hour, and the average query timepoint is 428.89 seconds.

4. Experiments

Experiments show that offline Video-LLMs can transfer some capabilities to real-time perception, but current models still struggle with temporal prioritization, hallucination control, forward responding, and efficient inference.

  • Main findings: Offline Video-LLMs perform competitively on Real-Time Visual Perception tasks, partially bridging offline and online video understanding.Their offline video comprehension capabilities can transfer to certain online scenarios.
  • Main findings: 58.43% on [STU] and 66.97% on [ACR] are achieved by the best proprietary models, leaving a significant gap compared to Human Agents.The models often fail to locate the correct scene when multiple misleading scenes match the question.
  • Main findings: 52.69% vs. 91.37% on [HLD] shows a hallucination gap between proprietary models and human performance.Hallucinations are particularly significant in open-source and online models.
  • Main findings: Around 4 seconds of average response latency at 64 frames keeps real-time video dialogue far from reach for efficient models such as QWen2VL-7B and FlashVStream.Inference latency grows exponentially as the number of input frames increases.
  • Online versus offline models: Gemini 1.5 Pro achieves the highest average score among the evaluated offline models, while Flash-VStream-7B performs comparatively lower in real-time perception tasks.The comparison indicates a potential trade-off between real-time processing constraints and depth of visual understanding.
  • Evaluation design: Forward Active Responding evaluates whether models can withhold responses until sufficient future evidence appears, while Backward Tracing and Real-Time Visual Perception expect immediate responses.The evaluation pipeline queries models densely along temporal axes to encourage timely clue finding and reduce hallucinations before clues appear.

5. Conclusion and Future Work

OVO-Bench evaluates online video understanding across backward tracing, real-time perception, and forward active responding. Its evaluation pipeline densely queries models over time, while metric design balances correctness, timeliness, and hallucination avoidance.

  • OVO-Bench assesses Backward Tracing, Real-Time Visual Perception, and Forward Active Responding as three modes of online video understanding.
  • Forward Active Responding: Offline models are densely queried along temporal axes to decide independently whether current visual content provides enough clues to answer.This multiple-triggering pipeline adapts offline models to the forward active responding setting.
  • Evaluation Pipeline and Metrics: Accuracy-based metrics average answer correctness, while score-based metrics additionally reward accurate and timely responses.
  • Evaluation Tasks: For Backward Tracing and Sequential Steps Recognition, queries are made at the video start; Clues Reveal Responding instead queries before every reference answer.
  • Prompt Design: Streaming prompts include accurate timestamps, while offline models receive clips from the beginning through the query time for evaluation.

7. More Details of Benchmark Construction

Benchmark construction combines automated question generation with volunteer-proposed questions and human refinement. Annotators identify discontinuities, reveal clues, and select timestamps when sufficient visual information becomes available.

  • Real-Time Visual Perception: Human labor supplements automatic generation for Real-Time Visual Perception tasks to reduce bias and increase question diversity.
  • Prompts and Examples: Prompt examples cover Optical Character Recognition and Action Sequence Identification alongside the real-time perception evaluation materials.
  • Question Construction: Annotators select moments with spatial relationships, notable objects, or unusual attributes, then construct visually relevant options and misleading distractors.
  • Clue Reveal Responding: For Clue Reveal Responding, volunteers identify apparent discontinuities and continue watching until enough visual information reveals the answer.
  • Clue Reveal Responding: Annotators record concise clue timestamps at the moment when sufficient evidence has just been revealed.

8. Additional Dataset Analysis

Additional analysis visualizes how questions, videos, query timestamps, and durations are distributed across OVO-Bench’s twelve tasks. The benchmark’s diversity remains constrained by data, generation, and annotation limitations.

  • Dataset Distributions: Figures 10 and 11 show distributions of questions, videos, averaged query timestamps, and video durations across the twelve OVO-Bench tasks.
  • Dataset Distributions: 6,857 seconds is the averaged video duration reported for the CRR task.
  • Limitations: OVO-Bench’s diversity is hindered by scarce suitable datasets, unsatisfactory automatic question generation, and high human annotation costs.The authors note that these constraints can cause potential bias.
  • Offline Models: Offline models may randomly guess in Forward Active Responding, and always answering “Yes” can still produce a score above zero.

10. Licenses

OVO-Bench annotations are distributed under CC BY-NC-SA 4.0, alongside the applicable licenses of source datasets and self-crawled video downloads.

  • Licensing: OVO-Bench annotations are provided under the CC BY-NC-SA 4.0 license.
  • Licensing: Downloading the dataset requires adherence to both the OVO-Bench license terms and the licenses of source datasets.
  • Licensing: Download links are provided for self-crawled YouTube videos, while source-dataset licenses are listed separately.

QA-Ego4D

This section presents timestamped questions about objects, actions, attributes, and spatial or positional facts in household videos.

  • Timestamped questions ask where objects were located before pickup, what was selected, and what action occurred.
  • The examples include binary event judgments, such as whether a drawer was left open.
  • One question asks whether the depicted home is on the first or second floor.
  • Another question identifies a gold object on a nightstand from four visual alternatives.

Open-EQA

This section contains timestamped visual questions about locations, colors, and possible answers, including cases where the answer may be unavailable.

  • Questions ask where a person can sit and eat and what color a smoke detector is.
  • The answer choices include alternative locations such as a kitchen bar counter, hallway floor, bed, and living-room couch.
  • Hallucination Detection includes questions with an explicit “Unable to answer” option.

QA-Ego4D

This section provides timestamped questions about the location of an object and the timing of related clues.

  • A question asks where the vacuum cleaner head was placed, with closet, bathroom, kitchen, and “Unable to answer” as options.
  • The accompanying timing information lists clue times of 7:10 and 7:30.

Open-EQA

This section spans tasks for hallucination detection, action sequencing, attribute and spatial recognition, OCR, preparation prediction, and repetition counting, using timestamped questions and answer choices.

  • Action Sequence Identification: Action-sequence questions ask what happened before or after a specified cooking, crafting, or object-handling action.
  • Action Sequence Identification: Several sequence questions include query, reference, or clue times to anchor temporal relationships.
  • Spatial Understanding: Spatial Understanding asks about a person's position relative to a car and which container is nearest the table's top-left corner.
  • Attribute Recognition: Attribute Recognition questions target weapon identity, material, tie appearance, clothing color, and the action performed with a cloth.
  • Optical Character Recognition: OCR questions ask for displayed text, including a leading player's time at the 150m mark and labels on packages or products.
  • Preparation-oriented questions ask what object or action a person is about to manipulate or perform.
  • Repetition Event Count: Repetition Event Count asks how many times an action occurred, exemplified by counting clean-and-jerk repetitions.

Thumos 14/15 Perception Test

The section illustrates timestamped video perception through sequential-step recognition, clue-based responding, and queries about actions, counts, destinations, and observed entities. Examples require answers to depend on events distributed across the video timeline.

  • Temporal Counting: Timeline queries test cumulative event counting by asking how many times an action occurred at progressively later timestamps.One example counts camera-facing displays as 1, 2, 3, 4, and 4 across query times; another counts removals as 0, 1, 2, 3, 4, 4, and 4.
  • Sequential Steps Recognition: Sequential Steps Recognition links timestamped clues into ordered procedures such as reserving hair space before applying extensions.The examples identify the two steps at 0:25–0:30 and 0:31–0:49.
  • Procedure Understanding: Procedure-understanding examples ask for complete instructions assembled from multiple timestamped cooking or craft actions.Examples cover hair extensions, omelet preparation, and sugar-coated haws using clues distributed across each video.
  • Clues Reveal Responding: Clue-based responding uses evidence revealed later than the query, such as answering a 2:25 question from a clue at 3:25.The examples include identifying why women enter the center door and what a man does with books.
  • Moment-Specific Understanding: The section also tests questions about what people see, do, or encounter at specified moments in longer videos.Examples ask about a policeman entering a wooden house, a woman entering a room, and people encountered while sitting on trunks.
Loading 2501.05510v2…