PaperlayerView exact revision on arXiv

Source-linked AI summary

QVHighlights: Detecting Moments and Highlights in Videos via Natural Language Queries

Jie Lei, Tamara L. Berg, Mohit Bansal

arXiv:2107.09609v2cs.CVcs.AIcs.CL

TL;DR

Long videos make it useful to locate query-relevant moments and highlights, but datasets have lacked comprehensive annotations for both tasks. The paper introduces QVHIGHLIGHTS and Moment-DETR, an end-to-end model for joint retrieval and saliency prediction. The authors report competitive baseline performance and substantially better results with weakly supervised ASR-caption pretraining.

  • Problem

    Query-based moment retrieval and highlight detection are under-studied, with limited datasets that comprehensively annotate both tasks for flexible user queries.

    Source: §1 · pp. 1–2

  • Method

    The paper builds QVHIGHLIGHTS with free-form queries, relevant moments, and clip-wise saliency ratings, and proposes Moment-DETR for end-to-end moment and saliency prediction.

    Source: §1 · p. 2, §4 · p. 5

  • Results

    Moment-DETR performs competitively with strong methods, and weakly supervised ASR-caption pretraining substantially outperforms previous methods.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    QVHIGHLIGHTS provides a unified benchmark for query-based moment retrieval and highlight detection, with annotations for diverse videos and queries.

    Source: §1 · p. 2

    System predictions reflect the collected dataset’s distribution and may be inaccurate, so users should not rely on them completely for real-world decisions.

    Source: §6 · p. 10

Abstract

from arXiv · show

Detecting customized moments and highlights from videos given natural language (NL) user queries is an important but under-studied topic. One of the challenges in pursuing this direction is the lack of annotated data. To address this issue, we present the Query-based Video Highlights (QVHIGHLIGHTS) dataset. It consists of over 10,000 YouTube videos, covering a wide range of topics, from everyday activities and travel in lifestyle vlog videos to social and political activities in news videos. Each video in the dataset is annotated with: (1) a human-written free-form NL query, (2) relevant moments in the video w.r.t. the query, and (3) five-point scale saliency scores for all query-relevant clips. This comprehensive annotation enables us to develop and evaluate systems that detect relevant moments as well as salient highlights for diverse, flexible user queries. We also present a strong baseline for this task, Moment-DETR, a transformer encoder-decoder model that views moment retrieval as a direct set prediction problem, taking extracted video and query representations as inputs and predicting moment coordinates and saliency scores end-to-end. While our model does not utilize any human prior, we show that it performs competitively when compared to well-engineered architectures. With weakly supervised pretraining using ASR captions, MomentDETR substantially outperforms previous methods. Lastly, we present several ablations and visualizations of Moment-DETR. Data and code is publicly available at https://github.com/jayleicn/moment_detr

1 Introduction

Long videos make whole-video search insufficient when users want query-relevant moments or highlights. QVHIGHLIGHTS combines annotations for both tasks, and Moment-DETR provides an end-to-end baseline for detecting them.

1 Introduction

  • Existing moment-retrieval datasets often have temporal bias and annotate only one moment per query, while many highlight datasets are query-agnostic or sparsely annotated.The two tasks are also typically studied separately because datasets rarely annotate both.Source: §1 · pp. 1–2
  • QVHIGHLIGHTS combines moment retrieval and query-based highlight detection annotations across more than 10,000 diverse YouTube videos.It supplies free-form queries, one or more relevant moments, and five-point saliency scores for every query-relevant 2-second clip.Source: §1 · p. 2
  • Moment-DETR treats retrieval as end-to-end set prediction and adds a saliency-ranking objective, avoiding manually designed pre- and post-processing.It is competitive with highly engineered architectures and substantially outperforms strong methods with weakly supervised ASR-caption pretraining.Source: §1 · p. 2
  • The paper reports that weakly supervised pretraining lets Moment-DETR substantially outperform several baselines on QVHIGHLIGHTS and CharadesSTA.The authors also present dataset analyses, model ablations, and visualizations.Source: §1 · p. 2

2 Related Work

Prior work treats moment retrieval and highlight detection with distinct datasets and methods. QVHIGHLIGHTS unifies their query-based study, while Moment-DETR predicts moments and saliency end-to-end.

Datasets and Tasks

  • Moment-retrieval datasets often show temporal bias and provide one moment per query, while most highlight datasets are query-independent and existing query-based sets annotate sparsely.QVHIGHLIGHTS instead provides multiple possible moments and five-point saliency scores for all query-relevant clips, supporting a unified benchmark.Source: §2 · p. 3

Methods

  • Moment retrieval approaches score proposals, predict start-end indices, or regress coordinates, while highlight methods typically rank frames or clips.Many retrieval approaches rely on hand-crafted proposal generation or non-maximum suppression, unlike Moment-DETR’s direct set prediction.Source: §2 · p. 3
  • Moment retrieval and highlight detection share query-video similarity needs, but are commonly addressed separately; Moment-DETR directly outputs moment coordinates and saliency scores from video and query representations.Its end-to-end design removes manually designed pre- and post-processing steps.Source: §2 · p. 3

3 Dataset Collection and Analysis

QVHIGHLIGHTS was collected from diverse YouTube lifestyle and news videos, with worker-written queries, relevant moments, and clip-level saliency ratings. Its annotations support varied, multi-moment retrieval and query-dependent highlight evaluation.

3.1 Data Collection

  • Videos were sourced from lifestyle vlogs and news raw footage, then segmented into 150-second clips for annotation.The source videos were 5–30 minutes long and included daily activities, travel, and serious world events.Source: §3.1 · p. 3
  • Workers selected all relevant 2-second clips for their written query, with multiple disjoint moments allowed for each query-video pair.The protocol allowed finer annotation than 5-second clips.Source: §3.1 · p. 3
  • Around 90% of checked query-video pairs had average inter-worker moment IoU above 0.9, indicating high agreement in the moment annotations.Three workers independently annotated moments for each of 600 query-video pairs used in the agreement check.Source: §3.1 · p. 4
  • Annotators rated every relevant clip on a five-option Likert scale, with three workers’ ratings retained because highlightness can be subjective.The scale ranged from “Very Good” to “Very Bad”; workers rated all selected clips rather than choosing only a few highlights.Source: §3.1 · p. 4

3.2 Data Analysis

  • The collection comprises 10,310 queries, 18,367 moments, and 10,148 videos across daily vlog, travel vlog, and news categories.The category-specific query vocabulary reflects daily activities in vlogs and more serious activities in news.Source: §3.2 · p. 4, Table 1 · p. 4
  • Moment lengths vary widely, while their centers are nearly evenly distributed across videos; QVHIGHLIGHTS averages 1.8 moments per query.About 38% of moments are at most 10 seconds, and about 23% exceed 30 seconds; the distribution suggests less temporal bias than prior datasets.Source: Figure 2 · p. 4, §3.2 · p. 5
  • Unlike query-independent or sparsely annotated highlight datasets, QVHIGHLIGHTS scores all relevant clips on a five-point scale for query-dependent evaluation.Table 2 compares existing moment-retrieval datasets in its top portion and highlight-detection datasets in its bottom portion.Source: §3.2 · p. 5, Table 2 · p. 5

4 Methods: Moment-DETR

Moment-DETR jointly localizes query-relevant video moments and predicts clip saliency using an end-to-end transformer encoder-decoder. Its design combines direct set prediction, matched localization and saliency losses, and weakly supervised pretraining from ASR captions.

4 Methods: Moment-DETR

  • Moment-DETR treats query-based moment localization as direct set prediction and jointly outputs moment spans and clip-wise saliency scores.The model uses learned moment queries and models global temporal relations between video clips.Source: §4 · p. 5

4.1 Architecture

  • Video and text features are projected into a shared space and encoded with positional encodings; decoder moment queries feed heads for saliency, moment coordinates, and foreground/background labels.Video features are extracted every two seconds using SlowFast and CLIP; query features use the CLIP text encoder.Source: §4.1 · pp. 5–6, Figure 3 · p. 6

4.2 Matching and Loss Functions

  • Hungarian matching assigns predictions to ground-truth moments; training combines L1 and 1D temporal-IoU localization, hinge saliency ranking, and down-weighted background classification.Saliency ranking contrasts high- and low-scoring clips within ground-truth moments and clips inside versus outside them.Source: §4.2 · pp. 6–7

4.3 Weakly-Supervised Pretraining via ASR

  • For weakly supervised pretraining, the model learns timestamp prediction from 236K YouTube ASR caption-timestamp pairs across 5,406 training videos.The pretraining setup removes the first saliency-loss term because saliency annotations are unavailable.Source: §4.3 · p. 7

5 Experiments and Results

Experiments evaluate Moment-DETR on moment retrieval and highlight detection, then analyze its losses, pretraining, query slots, and predictions. Against XML+, it improves lower-IoU retrieval scores, and weakly supervised pretraining improves results on both tasks.

5.1 Experimental Setup

  • QVHIGHLIGHTS uses 70/15/15 train, validation, and test splits; retrieval is evaluated with mAP and R@1, while highlight detection uses mAP and HIT@1.The model uses 10 moment queries and is trained for 200 epochs; pretraining runs for 100 epochs, and saliency scores are averaged across three annotators.Source: §5.1 · p. 7

5.2 Results and Analysis

  • Moment-DETR exceeds XML+ by >7% absolute on R1@0.5 and mAP@0.5, but scores lower on R1@0.7 and mAP@0.75; ASR-caption pretraining outperforms baselines on both tasks.Moment-DETR performs similarly to XML+ on highlight detection, while CLIP has the best HIT@1 despite much lower overall performance.Source: §5.2 · p. 8, Table 3 · p. 8
  • Removing saliency loss significantly reduces highlight-detection and moment-retrieval performance, while removing either L1 or IoU loss hurts retrieval but leaves highlight detection similar.The authors hypothesize saliency loss directly supervises query-clip similarity; Table 4 reports these loss ablations on the validation split with models trained from scratch.Source: §5.2 · pp. 8–9, Table 4 · p. 9
  • Across 1,550 validation videos, the 10 decoder slots learn varied temporal locations and widths; most predict short moments, which the authors link to the dataset's short-moment prevalence.Some slots favor short moments near video beginnings or ends, while others predict short and long moments near the center.Source: §5.2 · p. 9, Figure 4 · p. 9
  • 5.4K in-domain QVHIGHLIGHTS videos outperform 10K HowTo100M videos; more HowTo100M data steadily helps retrieval, while highlight-detection performance fluctuates.Carefully tuned XML+ pretraining still underperforms its no-pretraining version; the authors suggest its cross-entropy loss penalizes mismatches in noisy data.Source: §5.2 · pp. 8–9, Table 5 · p. 9
  • On CharadesSTA, Moment-DETR improves R1@0.5 but is slightly worse at R1@0.7; pretraining on 10K HowTo100M videos sets a new state of the art.The authors describe this result as evidence of potential to adapt to different data and tasks.Source: §5.2 · p. 10, Table 6 · p. 10
  • A visualization shows Moment-DETR localizing two disjoint query-relevant moments with saliency scores aligned to annotator averages, alongside a failure to distinguish basketball from other court play.Figure 5 marks predictions with solid red boxes or lines and ground truth with dashed green lines.Source: §5.2 · p. 10, Figure 5 · p. 10

6 Conclusion

The paper introduces QVHIGHLIGHTS and Moment-DETR for query-based moment retrieval and highlight detection. It reports stronger performance with ASR-caption pretraining and cautions against relying on predictions for real-world decisions.

6 Conclusion

  • QVHIGHLIGHTS contains over 10,000 diverse YouTube videos annotated with free-form queries, relevant moments, and clip-wise saliency scores.The paper also presents Moment-DETR, an encoder-decoder transformer for jointly performing moment retrieval and highlight detection.Source: §6 · p. 10
  • With weakly supervised pretraining using ASR captions, Moment-DETR substantially outperforms previous methods.The paper also reports competitive performance against baseline methods and effective learning from noisy data.Source: §6 · p. 10
  • The authors caution that predictions reflect the collected dataset's distribution, can be inaccurate, and should not be relied on completely for real-world decisions.Source: §6 · p. 10

A Additional Results

Additional results examine performance across video categories, the number of moment queries, saliency-loss terms, and qualitative predictions.

Performance breakdown by video category

Ablations on #moment queries

  • Using 10 moment queries gives better moment-retrieval performance; highlight-detection mAP changes little, while HIT@1 favors fewer queries.Considering both tasks, the authors use 10 moment queries for their best model.Source: A Additional Results · p. 11, Table 8 · p. 11

Saliency loss ablations

  • Adding either saliency-loss term improves performance across all metrics, the in/out-of-moment term typically works better, and using both achieves the best performance.The terms distinguish high- versus low-saliency clips and clips inside versus outside ground-truth moments.Source: A Additional Results · p. 11, Table 9 · p. 11

More prediction examples

B Additional Data Analysis and Collection Details

The additional data analysis reports saliency-score distributions and describes the two-stage annotation process and worker qualification.

Distribution of saliency scores

Annotation Instructions and Interfaces

C Content, License and Usage

The paper describes access, usage restrictions, and the collection and review context for the dataset.

Content, License and Usage

  • The data and code are publicly available, but the dataset is restricted to research use and must not be used for purposes such as surveillance that may violate human rights.Source: C Content, License and Usage · p. 12
  • The videos come from a curated list of non-offensive topics, and the authors state that they use them under Fair Use.Source: C Content, License and Usage · p. 12
  • The study used Amazon Mechanical Turk under university IRB approval, and the collected data contains no personally identifiable information.Source: C Content, License and Usage · p. 12

Related papers

Loading 2107.09609v2…