PaperlayerView exact revision on arXiv

Source-linked AI summary

Localizing Moments in Video with Natural Language

Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, Bryan Russell

arXiv:1708.01641v1cs.CV

TL;DR

Natural-language moment localization requires identifying when a described event occurs, but existing datasets lack uniquely localizing referring expressions and often use narrow or edited footage. The paper proposes MCN and collects DiDeMo, with ablations showing that temporal context and both appearance and optical-flow features matter.

  • Problem

    Existing video-language datasets lack natural-language descriptions that uniquely localize moments and are often small, domain-restricted, edited, or focused on entertaining segments.

  • Method

    The paper proposes the Moment Context Network, which combines local, global, and temporal endpoint video features, and collects DiDeMo with localized moments and referring descriptions.

  • Results

    Ablations demonstrate the importance of temporal context features and the need for both appearance and optical-flow features.

  • Takeaways & Limitations

    MCN and DiDeMo establish an open-world setting for studying natural-language localization of video moments.

  • Takeaways & Limitations

    Because annotators do not always agree on segment boundaries, achieving 100% on all evaluation metrics is impossible.

Abstract

from arXiv · show

We consider retrieving a specific temporal segment, or moment, from a video given a natural language text description. Methods designed to retrieve whole video clips with natural language determine what occurs in a video but not when. To address this issue, we propose the Moment Context Network (MCN) which effectively localizes natural language queries in videos by integrating local and global video features over time. A key obstacle to training our MCN model is that current video datasets do not include pairs of localized video segments and referring expressions, or text descriptions which uniquely identify a corresponding moment. Therefore, we collect the Distinct Describable Moments (DiDeMo) dataset which consists of over 10,000 unedited, personal videos in diverse visual settings with pairs of localized video segments and referring expressions. We demonstrate that MCN outperforms several baseline methods and believe that our initial results together with the release of DiDeMo will inspire further research on localizing video moments with natural language.

1. Introduction

The paper frames natural-language moment localization as identifying a video's start and end points for a referring description, then proposes MCN and DiDeMo to address modeling and data limitations.

  • Motivation: Action, object, or attribute keywords may fail to uniquely identify a moment when the same elements recur across a video.The motivating example distinguishes the girl jumping after falling from other frames containing the girl or jumping.
  • Task: Natural-language moment localization identifies the start and end points corresponding to a description in a video.The task requires both language and video understanding and has applications in retrieving particular moments from long personal videos or stock-video libraries.
  • Gap: Existing natural-language video retrieval methods retrieve entire videos but do not identify when a moment occurs within them.The proposed solution learns a joint video-language model in which referring expressions and corresponding moment features are close in a shared embedding space.
  • Approach: Global video context and temporal position provide cues beyond features from a specific moment.Relative language such as “closest” requires context, while queries such as “starts the race” can favor moments near the beginning of a video.
  • Dataset: DiDeMo contains over 10,000 unedited videos with 3-5 description–moment pairs per video and diverse open-world content.A validation step was included to ensure descriptions uniquely localize their moments.
  • Contributions: The paper contributes MCN, which relies on local and global video features, and DiDeMo, which contains over 40,000 referring-description and localized-moment pairs.

2. Related Work

Prior work addresses whole-video retrieval, constrained temporal retrieval, summarization, description, question answering, and spatial grounding, but does not meet the requirements of open-world temporal moment localization.

  • Video Retrieval with Natural Language: Natural-language video retrieval learns joint video-language embeddings to retrieve a specific video, whereas this task must also identify when an event occurs within it.The paper adds local, global, and temporal endpoint features for this temporal localization problem.
  • Temporal Segment Retrieval: Existing temporal-segment methods use constrained settings such as fixed spatial prepositions, 21 dashboard-camera videos, or structured instruction ordering.These constraints differ from the paper's open-world localization setting.
  • Video Summarization: Video summarization datasets and methods generally select important content rather than localizing moments specified by referring natural language.Some summarization work uses titles, category labels, or textual descriptions, but the related datasets do not fit the paper's requirements.
  • Video Description and QA: Video description and question answering pair videos with generated descriptions or aligned questions and answers, rather than uniquely referring expressions for temporal localization.
  • YouTube and Movies: YouTube and movie datasets are unsuitable because selected clips may lack referring descriptions and editing can remove uneventful content.A description such as “A woman is talking” may apply to an entire video rather than uniquely identifying one clip.
  • Language Grounding: Natural-language object retrieval localizes phrases spatially in images, while video-moment localization additionally requires temporal indicators and activity understanding.Prior video grounding methods also constrain vocabulary or visual domains.

3. Moment Context Network

MCN embeds language and temporal video context jointly, combining local content, global context, and endpoint timing, and trains with intra- and inter-video ranking losses.

  • Model: MCN represents a proposed interval with visual temporal context features and encodes the sentence with an LSTM before comparing them in a joint model.At test time, it optimizes a distance over the sentence, video, and candidate temporal interval.
  • Visual Temporal Context Features: Temporal context features integrate local features for moment content, global features for video context, and endpoint features for a moment's temporal position.The endpoint coordinates are normalized to [0, 1].
  • Visual Inputs: MCN uses both RGB appearance and optical-flow inputs to capture objects, attributes, and activities.RGB features are extracted with VGG pretrained on ImageNet, while optical flow supplies motion information.
  • Visual Temporal Context Features: The temporal context network combines local, global, and endpoint features through a two-layer ReLU network with separate weights for RGB and flow inputs.
  • Language Features: The language branch encodes sentences with an LSTM and projects its final hidden state through a fully connected layer.Dense GloVe word embeddings were preferred to one-hot encodings because the dataset remains relatively small.
  • Ranking Loss: The ranking loss compares each referring expression with intra-video and inter-video negatives so the model learns subtle distinctions and broader semantic differences.Inter-video negatives use matching start and end points to encourage semantic rather than temporal discrimination.

4. The DiDeMo Dataset

DiDeMo addresses the lack of large-scale datasets pairing referring expressions with localized moments by collecting diverse, unedited personal videos and validating annotations for unique temporal reference. Its annotations capture temporal, camera, spatial, and action-related language across varied moments.

  • Annotation and validation: A two-phase process pairs each description with a single distinct moment by collecting descriptions first and validating their temporal localization afterward.A moment is accepted when at least three of four annotators agree, with endpoints differing by at most one gif.
  • Annotation and validation: DiDeMo’s validation step distinguishes it from other video-language datasets by ensuring descriptions are referring expressions for specific video moments.The dataset comparison identifies this verification procedure as a distinguishing feature of DiDeMo.
  • Language characteristics: Annotations can describe camera movements and viewer-relative actions, reflecting the dataset’s focus on distinct moments rather than only objects.Examples include a camera panning to friends, zooming in on a baby, or a boy running toward the camera.
  • Language characteristics: DiDeMo contains more camera and temporal words than comparison datasets, while verbs are more common than in natural language object-retrieval datasets.Examples include camera terms such as “zoom” and “pan” and temporal indicators such as “after” and “first.”
  • Annotated time points: 72.34% of descriptions correspond to one five-second segment, 22.26% to two contiguous segments, and 86% of videos contain multiple distinct moments.Annotators selected an average of 2.57 distinct moments per video, with more moments occurring near video beginnings.

5. Evaluation

The evaluation compares MCN with retrieval baselines and analyzes which temporal and visual features improve localization on DiDeMo. Qualitative results show that MCN can localize diverse queries and refine predictions beyond five-second annotation segments.

  • Metrics: Evaluation accounts for disagreement among four human annotations by assigning each prediction its maximum score across annotation subsets.Because annotators differ on start and end points, perfect scores are impossible on all metrics.
  • Baseline Comparison: MCN outperforms the moment frequency prior and is compared with CCA and natural language object retrieval baselines on DiDeMo.CCA performs substantially better on mIoU than the other trained baselines, while MCN performs substantially better than the moment frequency prior.
  • Ablation Studies: Global video features and temporal endpoint features both considerably improve moment-retrieval performance.The ablations also find that combining appearance and optical flow features is important for best performance.
  • Qualitative Results: MCN localizes diverse descriptions involving temporal indicators such as “first” and camera motion such as “camera zooms”.These qualitative examples demonstrate localization across different language and visual cues.
  • Fine-grained Moment Localization: Fine-grained test-time proposals let MCN localize moments within five-second ground-truth chunks.The model correctly localizes “A ball flies over the athletes” near the start of the first temporal segment and another query before the fifth segment ends.

Supplemental

The supplemental material provides qualitative analyses, ablations, retrieval examples, annotation information, and discussion of annotation ambiguity and metrics.

  • Supplemental Analyses: The appendix includes qualitative examples of global and temporal endpoint features, RGB and optical-flow inputs, and the full MCN.It also includes an ablation of the inter-intra negative loss and results without language features.
  • Additional Experiments: Additional supplemental material covers number-word generation for Table 2 and a qualitative video-retrieval experiment.The appendix also points to a video example for the retrieval experiment.
  • Annotation Ambiguity: A supplemental discussion addresses ambiguity in annotations and the metrics used to evaluate predictions.The annotation ambiguity is illustrated by differing judgments about when “The woman leaves the room” begins.
  • Annotation Materials: The appendix includes a histogram of the moments marked by annotators and an annotation-tool demonstration.These materials document annotation behavior and the collection interface.

A. Impact of Global Video Features and TEF Features

Global video context and temporal endpoint features improve localization for queries whose meaning depends on temporal comparison or onset. RGB and optical flow provide complementary information, motivating their fusion in MCN.

  • Impact of Global Video Features: Global context improves localization of comparative or sequence-dependent queries such as “closest,” “first,” and “reaches the top”.Determining when a car is closest requires viewing the entire video, while context also helps identify the second baby’s first appearance and the dog reaching the stairs’ top.
  • Impact of TEF Features: Temporal endpoint features help distinguish when an event begins rather than merely recognizing that the event occurs.Without tef, the model retrieves people or a moving train but misses their first appearance or the train’s beginning to move.
  • RGB and Flow Input Modalities: RGB input favors appearance-based queries, whereas optical flow favors motion- and camera-motion-based queries.Examples include identifying a child, a man’s yellow shirt, a dog jumping at the camera, and a camera zooming on a drummer.
  • RGB and Flow Input Modalities: The fusion model uses the complementary nature of RGB and optical flow for best results.Both individual networks can correctly localize some moments, but their combination is used to improve performance.

C. Qualitative Results for MCN

MCN qualitatively localizes natural-language moments, while baseline behavior varies with whether queries depend on objects, actions, or camera movement. The model still fails on difficult repeated-event queries.

  • MCN correctly localizes “the blue trashcan goes out of view” near approximately 14s using fine-grained temporal windows.The localization plot contains two peaks, with the higher peak corresponding to the trashcan leaving view and the other to its return.
  • The natural language object retrieval baseline is expected to work better for queries centered on specific objects than for action or camera-movement queries.Examples contrast “a man in a brown shirt runs by the camera” with “man runs towards camera with baby.”
  • MCN performs well on DiDeMo but fails to properly localize difficult repeated-event queries such as holding up a green board for the third time.
  • A newer object-retrieval model can outperform the earlier object-retrieval baseline on its original task yet perform worse on DiDeMo.The authors suggest sentence triplet parsing and differences in training data as possible reasons.
  • The image-retrieval model is adapted by matching dependency-linked sentence fragments to temporal regions rather than proposed image bounding boxes.

E. Inter-Intra Negative Loss

The experiments compare inter-negative, intra-negative, and combined inter-intra negative losses, finding that considering both negative types gives the best performance.

  • Table 4 reports MCN baseline comparisons in rows 1–8 and inter-intra negative-loss ablations in rows 10–12.
  • Considering both inter- and intra-negative examples is important for best performance.

F. Importance of Language Feature

Language features improve moment retrieval beyond temporal-context-only modeling, although visual cues alone provide performance above chance and the frequency prior. The analysis also reports the vocabulary and word categories used for a main-paper table.

  • 25.04, 75.23, and 36.12 are the temporal-context-only model’s R@1, R@5, and mIoU results, respectively.This model outperforms chance and the moment-frequency prior but underperforms the full model.
  • Natural language is important for achieving the best results on DiDeMo, where 86% of videos include multiple distinct moments.
  • The table analysis groups words into camera, temporal, and spatial categories.Examples include zoom and pan; first and after; and left and background.
  • The model vocabulary contains 7,785 words, compared with 329,274 total words in the dataset.

H. Video Retrieval Experiment

The video retrieval experiment retrieves moments from all test-set videos using their proximity to a text query in the shared embedding space. Retrieved moments are reported as semantically similar to the queries.

  • The model retrieves five moments closest to a text query from all videos in the test set.
  • The query “zoom in on baby” retrieves camera-zoom moments involving babies or young children, while “camera zooms in” retrieves camera-zoom moments without babies.
  • An example video retrieval demonstration is provided through an external video link.

I. Annotation Ambiguity

Moment annotations can be inherently ambiguous because annotators may disagree about temporal boundaries, and DiDeMo’s evaluation reflects partial agreement. The dataset’s moment distribution and qualitative comparisons also frame how model performance should be interpreted.

  • Annotation Ambiguity: Three annotators selected the fourth segment for “zoom in on man,” while another included the following fifth segment.The disagreement concerns whether the moment ends during the zoom or after the camera remains zoomed in.
  • Evaluation: A model retrieving both the fourth and fifth segments scores higher than one retrieving the third and fourth segments.The metric rewards overlap with the annotators’ preferred moment rather than requiring one universally agreed endpoint.
  • Evaluation: Some examples cannot receive a perfect score because no single retrieved moment aligns with every combination of annotator boundaries.The paper notes that qualitative ground-truth examples avoid this issue by requiring agreement from at least three annotators.
  • Model Behavior: Global video features improve retrieval for moments whose meaning depends on comparisons or events across time.Examples include identifying when a car is closest to the camera or when a dog runs toward it.
  • Model Behavior: Temporal endpoint features help distinguish an event from another occurrence of the same content, such as when people first appear or a train begins moving.RGB features favor object-recognition queries, whereas optical flow favors movement-related queries.
  • Dataset Distribution: Moments in DiDeMo tend to be short and occur near the beginning of videos.This distribution motivates comparison with a prior baseline based on the most common start and end points.
Loading 1708.01641v1…