Source-linked AI summary
Localizing Moments in Video with Temporal Language
Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, Bryan Russell
TL;DR
The paper addresses the challenge of localizing video moments described with temporal language, which requires reasoning about relationships among different video segments. It proposes MLLC, which treats context as a query-conditioned latent variable, and introduces TEMPO for controlled and human-language evaluation. MLLC outperforms prior work on TEMPO-TL, TEMPO-HL, and DiDeMo, while discussion results indicate that temporal reasoning remains imperfect.
Problem
Existing moment-localization datasets contain insufficient temporal language for reliably training and evaluating models that reason about temporal relationships in video.
Method
MLLC models video context as a latent variable conditioned on the video-query pair, and TEMPO provides template-based and human-language temporal annotations.
Results
MLLC outperforms prior work on TEMPO-TL, TEMPO-HL, and the original DiDeMo dataset.
Takeaways & Limitations
Query-conditioned context supports moment localization across simple and temporally complex queries, while TEMPO enables focused evaluation of temporal-language reasoning.
Takeaways & Limitations
Even when MLLC localizes context correctly, it does not always correctly localize temporal sentences, indicating that temporal reasoning remains an area for improvement.
Abstract
from arXiv · showhide
Localizing moments in a longer video via natural language queries is a new, challenging task at the intersection of language and video understanding. Though moment localization with natural language is similar to other language and vision tasks like natural language object retrieval in images, moment localization offers an interesting opportunity to model temporal dependencies and reasoning in text. We propose a new model that explicitly reasons about different temporal segments in a video, and shows that temporal context is important for localizing phrases which include temporal language. To benchmark whether our model, and other recent video localization models, can effectively reason about temporal language, we collect the novel TEMPOral reasoning in video and language (TEMPO) dataset. Our dataset consists of two parts: a dataset with real videos and template sentences (TEMPO - Template Language) which allows for controlled studies on temporal language, and a human language dataset which consists of temporal sentences annotated by humans (TEMPO - Human Language).
1 Introduction
Temporal language makes video moment localization require reasoning about relationships among actions and intra-video context. The paper proposes MLLC and introduces TEMPO to study this capability, reporting improved performance over prior work.
- Motivation: Queries with temporal language require models to reason about relationships between actions occurring in different video moments.For example, “talks after bending down” requires relating the target action to an earlier contextual action.
- Motivation: Relevant contextual moments vary by query, including before or after the target moment, making query-independent context insufficient.The paper identifies uncertainty in both the location and length of the context needed for reasoning.
- Approach: MLLC models video context as a latent variable conditioned on each video-query pair.This lets the model attend flexibly to contextual moments and unifies prior moment-localization approaches.
- Dataset: TEMPO contains real videos with template-based temporal sentences and human-provided temporal annotations.Its two parts are TEMPO-Template Language and TEMPO-Human Language.
- Results: MLLC outperforms prior work on TEMPO-TL, TEMPO-HL, and the original DiDeMo dataset.The result supports evaluating temporal-language reasoning across both controlled and human-authored queries.
2 Related Work
Prior work localized natural-language descriptions to video segments and studied temporal language or reasoning in related settings. This paper differs by targeting temporal grounding in longer real videos with both template and human-language queries.
- Video Moment Retrieval: Earlier moment-retrieval models score how well a natural-language query aligns with candidate video segments.Related systems include approaches by Gao et al. and Hendricks et al.
- Video Moment Retrieval: Existing datasets include localized or aligned descriptions, but their temporal-language coverage is limited.Charades uses semi-automatic alignment, while DiDeMo contains localized referring expressions.
- Temporal Language: This work learns temporal language from data rather than deriving an explicit logical framework for temporal prepositions.It focuses on temporal words used to refer to video content.
- Modeling Visual Relationships: The approach is analogous to modeling spatial relationships in images, but applies relational reasoning to temporal video moments.CLEVR provides a related example of systematic reasoning evaluation in visual question answering.
- Modeling Visual Relationships: Unlike short-video temporal-reasoning VQA datasets, this work uses 25–30-second videos and includes temporal grounding with human-language queries.TGIF-QA videos average 3.1 seconds and use multiple-choice temporal questions; TEMPO includes real visual inputs and localized language.
3 Moment Localization with Latent Context
MLLC localizes a query’s base moment by maximizing similarity over possible context moments, allowing context selection to depend on each query/video pair. The unified formulation supports different feature, similarity, context, loss, and supervision choices.
- Latent-context formulation: MLLC scores a base moment by maximizing similarity over all possible context moments, treating context as a latent variable.The formulation uses video and language features, a similarity function, and model parameters, and subsumes prior single-context formulations.
- Model components: The video representation concatenates visual features for the base and context moments with temporal endpoint features.Visual features are averaged over each temporal region, with separate RGB and optical-flow embeddings combined through late fusion.
- Model components: Temporal endpoint features encode when the base moment starts and ends, while context endpoint features indicate when contextual moments occur.The model experiments with adding endpoint features for context moments to represent temporal relationships between segments.
- Model components: The model compares distance-based, fused-feature, normalized fused-feature, and TALL similarity functions for video-language matching.Fused similarities use the Hadamard product, while TALL similarity concatenates video, language, product, and sum features before an MLP.
- Context choices: Context choices range from the entire video and moments immediately before or after the base moment to all possible moments as latent context.The latent-context option offers the greatest flexibility in contextual reasoning.
- Training and supervision: Training uses either an MCN ranking loss or the TALL alignment loss, with inter-video negatives yielding approximately one point on all metrics over intra-video negatives alone.The study also contrasts weakly supervised latent-context optimization with strongly supervised context training when ground-truth context is available.
4 The TEMPO Dataset
TEMPO addresses the shortage of temporal language in existing moment-localization datasets with template-based and human-authored data. Its human-language portion adds natural linguistic complexity, including pronouns and alternate descriptions of reference moments.
- Dataset construction: TEMPO contains Template Language and Human Language subsets built from the DiDeMo dataset.Template Language provides controlled sentences with known base and context moments, while Human Language supplies fully user-constructed temporal sentences.
- Motivation: Existing moment-localization datasets contain temporal words but not enough examples to reliably train and evaluate temporal-language models.TEMPO focuses on the four most common video-grounded temporal words: “before,” “after,” “then,” and “while.”
- TEMPO-Template Language: TEMPO-TL combines adjacent DiDeMo moments with templates using “before,” “after,” and “then.”The dataset uses two templates each for “before” and “after,” and one template for “then.”
- TEMPO-Human Language: TEMPO-HL captures linguistic complexity that templates do not, including temporal prepositions and cohesive structures such as coreference.Human-authored long sentences frequently use coreference to form more cohesive statements.
- Annotation protocol: Human-language annotations ask workers to describe a segment relative to a reference moment and use separate workers to verify that descriptions are localizable.Videos are segmented into five-second intervals following the DiDeMo annotation protocol.
- TEMPO-Human Language: In TEMPO-HL, 28.1% of sentences include pronouns, compared with 10.3% in the original DiDeMo dataset.Annotators also use wording that differs from the original base-moment description while referring to the same moment.
5 Experiments
Experiments evaluate MLLC and competing moment-localization models on TEMPO-TL, TEMPO-HL, and DiDeMo, focusing on temporal context and context localization. MLLC performs strongly, while results also show that better context localization and temporal reasoning remain important.
- Evaluation: Evaluation uses DiDeMo’s five-second segment protocol with R@1, R@5, and mIoU on TEMPO-TL, TEMPO-HL, and DiDeMo.Training combines DiDeMo with either TEMPO-TL or TEMPO-HL.
- Base model: The MLLC-Base model outperforms previous MCN and TALL models on the original DiDeMo dataset.The base-network study compares ranking losses, similarity functions, and variants of the prior methods.
- TEMPO-TL: On TEMPO-TL, MLLC with global context outperforms MCN and TALL across all sentence types, while before/after context helps localize corresponding temporal moments.Strong context supervision performs better than weak supervision, and context TEF is important for best performance.
- TEMPO-TL and TEMPO-HL: For “then,” global context is stronger on TEMPO-TL, whereas the full MLLC model is stronger on TEMPO-HL because human-language examples may require temporal ordering among repeated moments.The authors attribute this difference to template examples whose constituent moments often occur only once.
- Context analysis: Correctly localizing context improves MLLC performance for all three temporal word types, but global context decreases performance for “before” and “after.”Even with correctly localized context, all three sentence types retain substantial room for improvement.
- TEMPO-HL: TEMPO-HL performance is considerably lower than TEMPO-TL, indicating that the human-language dataset is harder.On TEMPO-HL, MLLC has the best performance across all metrics, especially mIoU, and before/after context beats global context for those words.
- Context analysis: Ground-truth context improves “before,” “after,” and “then,” showing that better context localization could improve these sentence types.The study also finds that MLLC reliably selects correct context fragments in qualitative examples.
- Discussion: The authors identify better temporal reasoning, context localization, and broader data sources as avenues for improving moment retrieval.They note that even correctly localized context does not always yield correct temporal-sentence localization and that visual content remains diverse.
Appendix
The appendix provides an example of how TEMPO-TL sentences are created and includes additional qualitative examples.
- Appendix: The supplemental material illustrates TEMPO-TL sentence construction and presents additional qualitative examples.
A TEMPO-Template Language
TEMPO-TL creates temporal-language queries from adjacent DiDeMo moments by combining their descriptions with temporal words.
- TEMPO-Template Language: A “then” sentence concatenates adjacent DiDeMo moments, while an “after” sentence references one moment relative to another.Figure 5 shows the original DiDeMo sentences and the constructed temporal query examples.
B Qualitative Examples on TEMPO-Human Language
Qualitative examples show MLLC localizing original and temporally modified queries, while using context moments that are generally accurate and temporally sensible. Failures arise when queries are especially complex or when a sensible context relationship still accompanies an incorrect target moment.
- MLLC correctly localizes the original DiDeMo sentence and temporal queries using “before,” “then,” and “after” for the same video.
- Across additional examples, the model generally localizes context moments accurately and sensibly for temporal sentences.
- Examples with “before” and “after” show the model using multiple context moments and relating localized moments temporally.
- Failure cases include a complex query with “then” and “before,” and an “after” query whose context relationship is sensible but whose target moment is incorrect.
- The qualitative set includes correctly localized moments for “then” and “while,” as well as examples from DiDeMo.