PaperlayerView exact revision on arXiv

Source-linked AI summary

MomentSeeker: A Task-Oriented Benchmark For Long-Video Moment Retrieval

Huaying Yuan, Jian Ni, Zheng Liu, Yueze Wang, Junjie Zhou, Zhengyang Liang, Bo Zhao, Zhao Cao, Zhicheng Dou, Ji-Rong Wen

arXiv:2502.12558v5cs.CVcs.AI

TL;DR

Long-video understanding lacks benchmarks that directly test whether task-relevant moments are accurately retrieved. MomentSeeker fills this gap with a long, diverse, hierarchically organized benchmark supporting text, image-conditioned, and video-conditioned queries. Experiments across generation-based and retrieval-based methods show substantial accuracy and efficiency challenges despite recent model improvements and task-specific fine-tuning.

  • Problem

    Existing benchmarks are limited by short videos, narrow task diversity, or end-to-end evaluation that does not directly assess accurate key-moment access.

    Source: §1 · p. 1

  • Method

    MomentSeeker constructs a long-video benchmark with diverse domains, global-, event-, and object-level tasks, multimodal queries, and fine-grained temporal annotations.

    Source: §1 · p. 2, §3.4 · p. 6

  • Results

    Experiments with generation-based and retrieval-based methods reveal significant challenges in long-video moment retrieval accuracy and efficiency despite recent MLLM improvements and task-specific fine-tuning.

    Source: §1 · p. 3, §5 · p. 10

  • Takeaways & Limitations

    MomentSeeker provides a specialized evaluation resource for developing and analyzing accurate, efficient long-video moment retrieval systems.

    Source: §1 · p. 3, §5 · p. 10

    Existing models perform worse on image- and video-conditioned retrieval than text-only retrieval, especially generation-based methods.

    Source: §4.3 · p. 9

Abstract

from arXiv · show

Accurately locating key moments within long videos is crucial for solving long video understanding (LVU) tasks. However, existing benchmarks are either severely limited in terms of video length and task diversity, or they focus solely on the end-to-end LVU performance, making them inappropriate for evaluating whether key moments can be accurately accessed. To address this challenge, we propose MomentSeeker, a novel benchmark for long-video moment retrieval (LMVR), distinguished by the following features. First, it is created based on long and diverse videos, averaging over 1200 seconds in duration and collected from various domains, e.g., movie, anomaly, egocentric, and sports. Second, it covers a variety of real-world scenarios in three levels: global-level, event-level, object-level, covering common tasks like action recognition, object localization, and causal reasoning, etc. Third, it incorporates rich forms of queries, including text-only queries, image-conditioned queries, and video-conditioned queries. On top of MomentSeeker, we conduct comprehensive experiments for both generation-based approaches (directly using MLLMs) and retrieval-based approaches (leveraging video retrievers). Our results reveal the significant challenges in long-video moment retrieval in terms of accuracy and efficiency, despite improvements from the latest long-video MLLMs and task-specific fine-tuning. We have publicly released MomentSeeker(https://yhy-2000.github.io/MomentSeeker/) to facilitate future research in this area.

1 Introduction

MomentSeeker addresses the lack of suitable benchmarks for evaluating whether long-video systems can accurately retrieve task-relevant moments. It combines long, diverse videos, varied task and query types, fine-grained annotation, and experiments showing persistent accuracy and efficiency challenges.

  • Existing benchmarks use short videos, simplistic caption queries, or end-to-end answer evaluation, leaving accurate long-video moment retrieval insufficiently assessed.Source: §1 · pp. 1–2
  • MomentSeeker evaluates task-oriented retrieval by testing whether moments relevant to generic LVU tasks, including causal reasoning, can be identified in long videos.Source: §1 · p. 2
  • Videos average more than 1200 seconds and span movie, sports, anomaly, and egocentric domains, broadening coverage of real-world scenarios.Source: §1 · p. 2
  • The benchmark covers global-, event-, and object-level scenarios, nine task types, and text-only, image-conditioned, and video-conditioned queries.Source: §1 · p. 2
  • Fine-grained timestamps and informed human annotation support precise, high-quality evaluation rather than reliance on noisy metadata or model-generated labels.Source: §1 · p. 2
  • Experiments with retrieval-based methods and MLLMs show that long-video moment retrieval remains challenging in both accuracy and efficiency.Source: §1 · pp. 2–3

2 Related Work

Prior long-video benchmarks assess end-to-end understanding but do not deeply verify temporal grounding, while earlier moment-retrieval datasets emphasize short or restricted settings. MomentSeeker extends evaluation toward diverse, multimodal, task-aware long-video grounding.

  • VideoMME, MLVU, and LongVideoBench evaluate long-video understanding end to end, overlooking whether models retrieve the moments supporting their answers.Source: §2.1 · p. 3
  • Synthetic V-NIAH and frame-focused LV-Haystack provide narrower tests because they lack real-world multimodal complexity or broad task and scenario diversity.Source: §2.1 · p. 3
  • MomentSeeker explicitly verifies grounding by requiring exact temporal moments, addressing cases where correct answers may result from guessing rather than visual evidence.Source: §2.1 · p. 3
  • Earlier moment-retrieval datasets usually use videos under three minutes, synthetic or domain-specific content, and limited query forms.Source: §2.2 · p. 3
  • MomentSeeker instead combines sports, movies, surveillance, and other domains with text, text-plus-image, and text-plus-video queries.Source: §2.2 · p. 4

3 MomentSeeker

MomentSeeker defines long-video moment retrieval around temporally grounded evidence and constructs a diverse, hierarchical benchmark with multimodal queries. Its annotation and metrics support both single- and multi-moment evaluation across global, event, and object tasks.

Task Definition

  • Long-video moment retrieval predicts timestamped moments P for an LVU query and evaluates their consistency with ground-truth moments G using similarity and integration functions.Source: §3.1 · p. 4

Dataset Construction

  • The dataset averages 1201.9 seconds, reaches approximately one hour, and spans egocentric, sports, movies, cartoons, and surveillance videos.Source: §3.2 · p. 4, Table 1 · p. 5

Task Taxonomy

  • The hierarchical taxonomy covers global tasks requiring extended causal, temporal, or spatial reasoning, event tasks targeting localized actions, and object tasks requiring fine-grained perception.Source: §3.3 · pp. 4–5
  • Object-level tasks include object recognition, localization, attribute classification, and OCR-based reasoning, generally emphasizing short durations and high spatial accuracy.Source: §3.3 · p. 5

Query and Annotation Design

  • Queries combine text-only, text-plus-image, and text-plus-video modalities, with text-only queries forming the majority at a 5:2:2 distribution.Source: §3.3 · p. 6, §3.4 · p. 7

Data Annotation

  • Answering moments are the minimal necessary and sufficient temporal evidence, allowing single segments for single-detail questions and multiple nonredundant segments for multi-detail questions.Source: §3.4 · p. 6

Evaluation Metrics

  • Recall@1 measures whether the top prediction overlaps any ground-truth moment, while mAP@5 evaluates accuracy and ranking among the top five segments.Source: §3.5 · p. 7

4 Experiments

Experiments compare retrieval-based and generation-based methods across tasks, query modalities, interval positions, and video lengths. Results show that long-video moment retrieval remains difficult, with retrieval methods generally stronger, generation methods benefiting from scale and context, and retrieval quality correlating with downstream LVU performance.

Main Results

  • 19.7 R@1 is achieved by InternVideo2, while Qwen2.5VL-72B reaches 17.2 R@1 and InternVL3-8B reaches 5.9 R@1, showing broadly weak moment-retrieval performance.E5V reaches 14.3 R@1, and the results attribute the difficulty to fine-grained temporal perception and complex multimodal reasoning.Source: Table 2 · p. 8, §4.2 · p. 9
  • 29.6 R@1 and 31.4 mAP@5 are achieved by Gemini 2.5 Pro, yet precise temporal localization remains difficult even for advanced MLLMs.The benchmark exposes weaknesses on queries requiring fine-grained localization, such as counting goals.Source: §4.3 · p. 9

Query Modalities

  • Most models perform worse on IMR and VMR than on TMR, with the gap especially large for generation-based methods.The result indicates difficulty integrating and reasoning across modalities in complex multimodal queries.Source: §4.3 · p. 9, Figure 5 · p. 9

Position and Duration

  • Generation methods are position-sensitive, whereas retrieval methods are largely position-insensitive; all models decline as video duration increases.Generation models often predict intervals near video boundaries, while retrieval models treat chunks more uniformly.Source: Figure 7 · p. 9, §4.3 · p. 10

Scaling and Context

  • 12.4 R@1 and 17.7 mAP@5 from 148M-parameter MM-Ret outperform InternVL3-8B’s 5.9 R@1 and 6.1 mAP@5, although larger generation models narrow the gap.Qwen2.5VL-72B performs better on short videos, while its 768-frame context corresponds to approximately 6.4 minutes at 2 fps.Source: §4.3 · p. 10

Downstream Validity

  • MomentSeeker retriever performance positively correlates with downstream RAG-based LVU results, indicating predictive value for LVU task capability.Different retrievers select top-k moments for the same InternVL3-38B model, linking retrieval quality to downstream performance.Source: Figure 6 · p. 9, §4.3 · p. 10

5 Conclusion

MomentSeeker is introduced as a benchmark for accurately locating key moments in long-video understanding. It covers diverse tasks and query types, while experiments reveal persistent accuracy and efficiency challenges.

Conclusion

  • MomentSeeker benchmarks key-moment localization for MLLMs across text-only, image-conditioned, and video-conditioned queries.The benchmark covers a wide range of real-world tasks and was released to support future research.Source: §5 · p. 10
  • Experiments show significant long-video moment-retrieval challenges in accuracy and efficiency despite recent MLLM and task-specific fine-tuning improvements.Source: §5 · p. 10

6 Appendix

The appendix documents compute, baseline reproduction, frame-sampling, and model-specific evaluation settings. These settings use official implementations and frame configurations tailored to the evaluated models.

Implementation

  • Experiments use 8×A800 GPUs with 80 GB of memory per GPU.Source: §6.1 · p. 14

Retrieval Evaluation

  • Baselines use official implementations and default settings, with LanguageBind and InternVideo2 evaluated on 8 sampled frames and COVR on 15 frames.Instructions guide query and candidate embeddings toward informative representations.Source: §6.1.1 · p. 14

Generation Evaluation

  • Generation-method evaluation uses official repository code and model-specific optimal frame counts from the original long-video papers.Source: §6.1.2 · p. 14

Model Settings

  • GPT-4o evaluation samples 180 test cases, extracts frames at 0.5 fps, and uniformly limits videos exceeding 128 frames to 128 frames.Frames are resized so the maximum image dimension does not exceed 384.Source: §6.1.2 · p. 14
  • Lita uses its default setting, extending videos by 100 frames without resizing and receiving timestamps for sampled frames.The timestamps allow the model to infer each frame’s temporal position.Source: §6.1.2 · p. 14

Time Instruction

The time instruction presents video duration and sampled-frame timestamps before task-specific prompts. TMR then asks the model to identify an event and return its start and end times.

Time Instruction

TMR TASK PROMPT

The TMR prompt asks models to identify a visual event in a video and return its start and end times. TimeChat uses uniformly sampled frames in its default setting.

TMR TASK PROMPT

  • The task requires locating the visual event described by a textual query or caption and reporting its start and end timestamps.The prompt specifies an output format such as “start time - end time.”Source: TMR TASK PROMPT · p. 14
  • TimeChat uniformly samples videos to 96 frames in the default TMR setting.Source: TMR TASK PROMPT · p. 14

TimeChat Query-Type TMR Task

The query-type TMR prompt asks TimeChat to find the video segment matching a sentence and output its temporal boundaries. The response uses a fixed start-time/end-time format.

TimeChat Query-Type TMR Task

TimeChat Caption-Type TMR Task

The TMR prompts define temporal retrieval as returning one or more relevant video intervals for a textual query or caption. Related prompts also support image- and video-conditioned retrieval.

TimeChat Caption-Type TMR Task

Annotation Guideline

The annotation guideline defines how to create and validate video-search tasks, including text-only, image-conditioned, and video-conditioned queries. It also records evaluation settings, limitations, and future directions.

Annotation Guideline

  • Annotators create questions answerable by one or more precise video segments and mark every necessary answering interval.Text-only questions and answers must originate from the same video, with replay used to validate alignment.Source: Annotation Guideline · pp. 15–16
  • Image-conditioned search links a selected frame to a question whose answer exists in the target video, including same-video and cross-video cases.Cross-video questions must preserve logical consistency between the image, question, and target segment.Source: Annotation Guideline · p. 16
  • Timestamp annotations require less than one second of error tolerance and must be checked for precise alignment.Source: Annotation Guideline · p. 16

6.2 Ablation of Different Video Frame Numbers

  • Increasing Qwen2.5VL-7B input frames from 96 to 768 raises R@1 from 4.5 to 8.1, but substantially increases memory and latency.InternVL3-38B remains stable from 64 to 128 frames, with R@1 values of 15.9, 15.8, and 15.6.Source: §6.2 · p. 16

6.3 Ablation of Different IoU Thresholds

  • Stricter IoU thresholds, including 0.5, reduce accuracy across models, while the main experimental conclusions remain consistent across thresholds.Source: §6.3 · p. 17

Broader Impacts

  • The benchmark uses publicly available videos that may remain incomplete or contain omissions despite prior harmful-content screening.Source: Broader Impacts · p. 18

7 Limitations & Future Works

  • The study mainly evaluates open-source models because processing GPT-4o at scale is costly, and it does not investigate model-specific temporal architectural choices.Processing 384 frames with GPT-4o costs approximately $2 per query.Source: §7 · pp. 18–19
  • Future work targets stronger temporal awareness in MLLMs and lightweight retrievers within RAG systems to improve efficiency.Source: §7 · p. 19

Related papers

Loading 2502.12558v5…