PaperlayerView exact revision on arXiv

Source-linked AI summary

MultiVENT-Raw: A Benchmark for Retrieval and Reasoning over Raw Videos

Reno Kriz, David Etter, Alexander Martin, Cameron Carpenter, Debashish Chakraborty, Hannah Recknor, Reihaneh Iranmanesh, Matthew Maciejewski, Kenton Murray, Eugene Yang, Benjamin Van Durme, Aaron Steven White, Andrew Yates, William Walden

arXiv:2609.28437v1cs.CVcs.IR

TL;DR

Raw video often lacks the contextual cues that help interpret edited footage, while many benchmarks focus on simple queries over edited videos. MultiVENT-Raw pairs a multilingual raw-video collection with event-centric retrieval and report-generation tasks; its baseline results show both tasks remain challenging.

  • Problem

    Raw video often lacks the contextual cues found in edited footage, while benchmarks have focused overwhelmingly on simple queries over edited videos.

    Source: §1 · p. 1

  • Method

    The paper builds a multilingual raw-video benchmark with event-centric queries, human relevance judgments, key facts for relevant videos, and retrieval and report-generation tasks.

    Source: §1 · pp. 1–2, §3 · p. 2

  • Results

    Strong retrieval baselines show middling performance, and generation models also perform underwhelmingly on MultiVENT-Raw.

    Source: §6 · p. 8

  • Takeaways & Limitations

    Raw-video retrieval is especially challenging when the collection mixes raw and edited video.

    Source: §6 · p. 8

    Hard-negative videos could not be manually verified as raw because of their large number.

    Source: §3.4 · p. 5

Abstract

from arXiv · show

Online information is increasingly consumed in video format. Much of this comes in the form of *raw video*: continuous footage taken on a cell phone, with a hand-held camera, or via CCTV, which is then directly uploaded to social media platforms and content sharing services. Whereas professional or even amateur-edited footage tends to feature scripted speech, chyrons, graphics, and metadata that help contextualize its subject matter, raw video typically contains none of these things, making it a much more challenging medium for information retrieval and machine understanding. To facilitate progress in this domain, we release MultiVENT-Raw, a multilingual collection of nearly 120,000 primarily raw videos (over 5,300 total hours), paired with 130 events and 222 event-centric queries, along with human-annotated video relevance judgments and human-extracted key facts for relevant videos. MultiVENT-Raw supports both a retrieval task---to identify videos in the collection relevant to a query event---and a generation task---to summarize event-related videos into a coherent report for a target user. We benchmark strong baselines on MultiVENT-Raw, showing both tasks to be challenging even for some of the latest multimodal models.

1. Introduction

Raw video often lacks the contextual cues found in edited footage, making high-level retrieval and understanding challenging. MultiVENT-Raw addresses this with a large multilingual benchmark and a smaller development set, alongside baseline results for retrieval and generation.

1. Introduction

  • Raw footage often lacks narration, overlays, captions, and metadata, leaving context implicit and requiring inference to match high-level queries to relevant video.Such footage includes continuous, unedited recordings made on cell phones, hand-held cameras, or CCTV.Source: §1 · p. 1
  • MultiVENT-Raw pairs nearly 120,000 videos across 130 events and 222 queries with manually annotated relevance judgments and key facts for relevant videos.The collection covers eight languages and more than 5,300 hours of footage.Source: §1 · p. 1
  • The release includes MICROVENT, a 23-event development set for rapid iteration, and baseline results showing both retrieval and generation remain challenging.Topics include anomalies drawn from livestream cameras sampled regularly over 11 months.Source: §1 · p. 2

2. Related Work

Prior video-retrieval benchmarks often use short edited clips and summary- or caption-derived queries, while multi-video report generation has only recently received meaningful attention. MultiVENT-Raw differs in its raw-video focus, broader language coverage, and analytic queries; related work also spans retrieval pipelines and emerging multi-video RAG approaches.

Text-to-Video Retrieval: Benchmarks

  • Many existing benchmarks use short, English-only edited clips and summary- or caption-derived queries, which can be bland and unlike real information needs [23] [43].Some multilingual-query benchmarks lack multilingual speech content in their videos.Source: §2 · p. 2
  • MULTIVENT 1.0 and MULTIVENT 2.0 [20] cover multilingual event-centric retrieval, but MultiVENT-Raw restricts relevant videos to raw content and adds Pashto, Farsi, and Urdu.MultiVENT 2.0 expanded the earlier benchmark with more queries and videos and finer-grained questions about event aspects.Source: §2 · p. 2

Text-to-Video Retrieval: Methods

  • Text-to-video retrieval commonly uses a two-stage pipeline: an efficient first-stage retriever selects candidates, then a more expressive reranker rescores them.Alternatives include text-derived video representations or separately encoded modalities with fused rankings.Source: §2 · p. 2

Multi-Video Retrieval-Augmented Generation

  • Multi-video RAG supports report generation from multiple videos, an area with recent work including knowledge-graph synthesis, text-based retrieval, and the WIKIVIDEO shared task.The paper evaluates CAG [26] and top participant systems TRACE [50] and MARQUIS [3].Source: §2 · p. 2

3. Data Collection

MULTIVENT-RAW defines a strict raw-video scope and builds a multilingual event collection with queries, relevance judgments, and claim annotations. Its curation covers primary events and livestream anomalies, while MICROVENT provides a separate development set.

3. Data Collection

  • MULTIVENT-RAW defines raw video as continuous, effectively unscripted and unedited footage, excluding scripted speech, narration, music, overlays, and multiple scenes.News broadcasts, amateur-edited videos, and compilations are filtered out; minimal logos or livestream timestamps are permitted in select otherwise-raw cases.Source: §3 · p. 2

3.1. Topic and Video Curation

  • 703 relevant videos cover 130 events, with 118,099 remaining collection videos serving as hard negatives.The events comprise 96 primary and 34 anomaly events. Anomalies were selected from livestream cameras sampled over 11 months after establishing each camera’s normal activity profile.Source: §3.1 · pp. 3–4

3.2. Query and Persona Development

  • Queries use personas to capture broad event-information needs or narrower questions, yielding 222 queries overall.Broad queries draw on claim annotations; narrow queries use personas and relevant-video titles and may ask questions the collection cannot answer. Anomaly events receive broad queries only.Source: §3.2 · p. 4

3.3. Claim Annotation

  • Annotators wrote persona-relevant English claims for each relevant video, recording confidence and the visual, on-screen text, speech, or non-speech audio supporting each claim.Claims were written after annotators watched videos for a target persona; multiple modalities could support a claim.Source: §3.3 · p. 5

3.4. Quality Control and Postprocessing

  • Irrelevant videos could not be manually verified as raw, so a classifier labeled them and defined FULL and CORE collection splits.The classifier achieved 91.4% dev-set F1; CORE removes irrelevant videos labeled edited, while FULL retains all original videos.Source: §3.4 · p. 5
  • Claim postprocessing made claims unique, atomic, and self-contained, then organized them into nested event, persona, and query relevance sets.Query claims were augmented with negative claims for query facets unanswered by the collection; these claims have no modality-support annotations.Source: §3.4 · p. 5

3.5. MICROVENT

  • MICROVENT is a separate development set released so MULTIVENT-RAW can serve as a test set.It contains 23 events, 31 queries, and 933 videos; 10 primary events come from MULTIVENT 2.0, and edited videos are not filtered out.Source: §3.5 · p. 5

4. Tasks

The paper evaluates retrieval by ranking videos for an event query and evaluates report generation by producing evidence-supported reports from retrieved videos. The generation task also has an oracle setting to assess performance independently of retrieval quality.

4.1. Video Retrieval

  • Retrieval ranks videos for an event query and is evaluated with recall@20/100 and nDCG@10/20.Recall measures the fraction of relevant videos retrieved in the top k; nDCG measures how closely the ranking approximates a perfect order.Source: §4.1 · p. 5

4.2. Report Generation

  • Report generation produces a query-focused report with video citations, evaluated for information recovery and citation quality in RAG and ORACLE settings.RAG uses a retrieval system’s ranked list; ORACLE uses the ideal ranked list to assess generation independently of retrieval quality.Source: §4.2 · p. 6

5. Experiments

Experiments evaluate segment-level retrieval and report generation on MULTIVENT-RAW’s FULL and CORE splits. Strong retrieval and generation baselines still struggle, and results vary with retrieval quality and split.

5. Experiments

  • Experiments chunk videos into segments of at most 5 minutes, indexing segments for retrieval and supplying them as VLM inputs.The resulting indexes contain 143,288 FULL and 75,147 CORE segments.Source: §5 · p. 6

5.1. Retrieval

  • OmniEmbed-mv, fine-tuned on MULTIVENT 2.0, achieves the best first-stage retrieval results by a wide margin; FULL is harder than CORE across metrics.Reranking consistently improves FULL, with Gemma 4 best; on CORE, it improves R@20 but has no consistent effect on nDCG.Source: §5.1 · p. 6

5.2. Report Generation

  • TRACElite achieves the highest Information F1 scores, while MARQUISbullet tends to achieve the best Citation F1 scores.Both TRACE and MARQUIS tend to outperform CAG.Source: §5.2 · p. 7
  • Information F1 differs by approximately 6–8 points between OmniEmbed-mv with reranking and OmniEmbed, demonstrating sensitivity to the retrieval system.Table 2 reports MiRAGE metrics for FULL and CORE with the top 10 videos from three retrieval systems.Source: §5.2 · p. 7, Table 2 · p. 7
  • Generation scores are generally modestly higher on CORE than FULL, though this pattern is not universal.The split pattern is consistent with first-stage retrieval results.Source: §5.2 · p. 8
  • Absolute report-generation scores are very low, especially citation metrics, indicating that the task remains difficult.The evaluation covers both information recovery and citation quality.Source: §5.2 · p. 8

6. Conclusion

MultiVENT-Raw benchmarks retrieval and report generation over raw, event-centric video, with baseline results showing both tasks remain challenging. The authors release MICROVENT to support rapid system development and intend MultiVENT-Raw to track progress in retrieval and reasoning over raw video.

6. Conclusion

  • The benchmark spans 130 events and nearly 120,000 videos, and includes a MICROVENT dev set of 23 events and close to 1,000 videos.The authors also present baseline results for retrieval and generation.Source: §6 · p. 8
  • Even strong dense-retrieval and recent reranking baselines show middling performance, with mixed raw and edited video posing particular retrieval difficulty.Generation baselines also struggle to recover key information and provide appropriate citations.Source: §6 · p. 8
  • The authors release MultiVENT-Raw to track future progress in retrieval and reasoning over raw video.They point to rapid improvements in multimodal models as grounds for expecting further advances.Source: §6 · p. 8

A. Data

The appendix describes a classifier used to derive the CORE split from FULL and outlines its development using MULTIVENT 2.0 labels. It also identifies the summary statistics reported for MICROVENT and MULTIVENT-RAW (FULL).

A.1. Summary Statistics

  • Table 3 reports summary statistics for MICROVENT and the FULL split of MULTIVENT-RAW.Source: §A.1 · p. 11

A.2. Raw Video Classification

  • The raw-video classifier is applied to FULL to create CORE by removing videos labeled as edited; training uses the MULTIVENT 2.0 test split, the only split with the four video-type labels.Those labels are raw, diet raw, edited, and professional; the binary scheme groups the first two against the latter two.Source: §A.2 · p. 11
  • The selected classifier is a weighted ensemble of five systems: an FFmpeg-based feature classifier and four Qwen-based prompted or fine-tuned classifiers.The components include two zero-shot variants and binary- and four-way-label LoRA models; the four-way model scores raw by summing raw and diet raw label log-likelihoods.Source: §A.2 · p. 11
  • The ensemble’s development used random 85%/15% train/dev splits and optimized binary Macro-F1 on the dev split.The raw score threshold was set at −0.657.Source: §A.2 · p. 11

B. Evaluation

The evaluation uses reference claims to assess generated reports and adapts claim processing for long predictions. Claim decomposition makes individual claims independently assessable, while recall is computed over fixed-size windows.

B. Evaluation

  • Reported MiRAGE metrics use the reference setting, comparing predicted-report claims with reference claims rather than assessing support directly against video content.Source: B. Evaluation · p. 11
  • Claim decomposition and decontextualization allow individual claims in long reports to be understood and evaluated without lengthy contextual passages.The approach adapts methods for claim decomposition and decontextualization.Source: B. Evaluation · p. 12
  • Because TRACE often produces reports of at least 10,000 sentences, precision uses claim decontextualization and recall uses windows of 100 claims.The windows together cover the predicted report’s decomposed and decontextualized claims.Source: B. Evaluation · p. 12

C. Generation

The appendix reports generation-baseline results in an ORACLE setting and additional results using top k=20 videos.

C. Generation

  • Table 5 reports ORACLE-setting baseline generation performance on MICROVENT and the CORE split of MULTIVENT-RAW.Source: C. Generation · p. 12
  • Table 6 reports baseline generation performance using the top k=20 videos instead of the top k=10 reported in main-text Table 2.Source: C. Generation · p. 12
  • Table 6 covers the FULL and CORE MULTIVENT-RAW splits and results from three retrieval systems.Source: C. Generation · p. 12

D.1. Gemma 4 Reranking

Gemma 4 zero-shot reranking scores query–video pairs independently using sampled video frames and a reranking prompt.

D.1. Gemma 4 Reranking

  • Gemma 4 scores each query–video pair independently, using eight uniformly sampled frames and yes–no log-odds from the first generated-token position.Each frame is capped at 280 visual tokens.Source: §D.1 · p. 12
  • Figures 4–6 provide prompts for Gemma 4 zero-shot reranking and the ZS v1 and ZS v2 variants.Source: Figure 4 · p. 12, Figure 5 · p. 12, Figure 6 · p. 12
  • Table 5 reports reference-variant MiRAGE results with Qwen3.5-27B as judge for ORACLE generation on MICROVENT and CORE.Source: Table 5 · p. 13
  • Table 6 reports reference-variant MiRAGE results for FULL and CORE using top k=20 videos from three retrieval systems.Source: Table 6 · p. 13
  • Figure 7 scores the generated binary label strings by log-likelihood, while Figure 8 sums raw and diet raw label log-likelihoods for binary prediction.Source: Figure 7 · p. 14, Figure 8 · p. 14
  • Figure 9 prompts claim decomposition; its purpose is to make claims independently understandable and evaluable without long contextual passages.Source: Figure 9 · p. 15
  • Figure 10 provides the reference claim set and a 100-claim predicted-report window for information verification.Source: Figure 10 · p. 16
  • Figure 11 runs citation verification once per cited source against a list of claims, then aggregates outcomes with logical OR.Source: Figure 11 · p. 16
Loading 2609.28437v1…