PaperlayerView exact revision on arXiv

Source-linked AI summary

Video Corpus Moment Retrieval with Contrastive Learning

Hao Zhang, Aixin Sun, Wei Jing, Guoshun Nan, Liangli Zhen, Joey Tianyi Zhou, Rick Siow Mong Goh

arXiv:2105.06247v1cs.CLcs.CVcs.IR

TL;DR

VCMR retrieves a query-matching moment from a corpus of untrimmed videos, but separate encoding is more efficient while cross-modal interaction often achieves better accuracy. ReLoCLNet adds video- and frame-level contrastive objectives to separate encoders, achieving accuracy comparable to cross-modal interaction baselines with much faster retrieval.

  • Problem

    VCMR must match text and video from distinct feature spaces, balancing the efficiency of separate encoding against the often-better retrieval accuracy of cross-modal interaction.

    Source: §1 · p. 1

  • Method

    ReLoCLNet adds VideoCL for matching video-query pairs and FrameCL for matching query-relevant moments within videos to a separately encoded retrieval network.

    Source: §1 · p. 2, §3.6 · p. 5

  • Results

    On two benchmarks, ReLoCLNet achieves accuracy comparable to cross-modal interaction learning, and on TVR it is about 56 times faster.

    Source: §1 · p. 2

  • Takeaways & Limitations

    ReLoCLNet offers comparable retrieval accuracy with much faster retrieval than cross-modal interaction baselines.

    Source: §1 · p. 2

    The presentation assumes videos have subtitles; without them, the subtitle encoding pipeline is removed and the final output is H_v only.

    Source: §3.1 · p. 3, §3.3 · p. 4

Abstract

from arXiv · show

Given a collection of untrimmed and unsegmented videos, video corpus moment retrieval (VCMR) is to retrieve a temporal moment (i.e., a fraction of a video) that semantically corresponds to a given text query. As video and text are from two distinct feature spaces, there are two general approaches to address VCMR: (i) to separately encode each modality representations, then align the two modality representations for query processing, and (ii) to adopt fine-grained cross-modal interaction to learn multi-modal representations for query processing. While the second approach often leads to better retrieval accuracy, the first approach is far more efficient. In this paper, we propose a Retrieval and Localization Network with Contrastive Learning (ReLoCLNet) for VCMR. We adopt the first approach and introduce two contrastive learning objectives to refine video encoder and text encoder to learn video and text representations separately but with better alignment for VCMR. The video contrastive learning (VideoCL) is to maximize mutual information between query and candidate video at video-level. The frame contrastive learning (FrameCL) aims to highlight the moment region corresponds to the query at frame-level, within a video. Experimental results show that, although ReLoCLNet encodes text and video separately for efficiency, its retrieval accuracy is comparable with baselines adopting cross-modal interaction learning.

1 INTRODUCTION

VCMR retrieves a query-matching moment from a corpus of untrimmed videos, a setting that differs from single-video moment retrieval and better matches corpus-based applications. ReLoCLNet uses contrastive learning to align separately encoded video and text, aiming to retain efficient retrieval while achieving accuracy comparable to cross-modal interaction baselines.

1 INTRODUCTION

  • VCMR retrieves a query-matching temporal fraction from a corpus of untrimmed, unsegmented videos, unlike single-video moment retrieval.The paper notes applications including query-based video surveillance, search, and navigation.Source: §1 · p. 1
  • Cross-modal interaction can improve retrieval accuracy but requires encoding each query against every corpus video, whereas unimodal encoding permits video pre-encoding.The paper adopts unimodal encoding for its efficiency and identifies encoder alignment during training as the challenge.Source: §1 · p. 1, Figure 1 · p. 2
  • Contrastive learning can refine unimodal encoders because both approaches emphasize relevant information in matching video-query pairs.The paper treats matching pairs as positive and non-matching pairs as negative.Source: §1 · p. 2
  • ReLoCLNet adds VideoCL and FrameCL to ReLoNet, targeting video-level matching and frame-level localization while allowing separate encoding after training.FrameCL treats features inside the target moment as foreground positives and the remaining features as background negatives.Source: §1 · p. 2
  • ReLoCLNet achieves accuracy comparable to cross-modal interaction baselines on two benchmarks and is about 56 times faster on TVR.The paper reports this as part of its main contribution.Source: §1 · p. 2

2 RELATED WORK

The related work spans video retrieval, single-video moment retrieval, VCMR, and contrastive learning. It distinguishes efficient unimodal approaches from fine-grained cross-modal interaction approaches and situates ReLoCLNet among them.

Video Retrieval

  • Video retrieval methods rank candidate videos by query relevance, with prior work exploring joint representations, sequence translation, hierarchical matching, and multiple encoders.The cited examples include [54],, [79], and [39], respectively.Source: §2 · p. 2

Single Video Moment Retrieval

  • Single-video moment retrieval work includes proposal-ranking, anchor-based, regression, span-based, and sequential decision-making approaches.The passage also notes weakly supervised learning and joint training with event captioning.Source: §2 · p. 2

Video Corpus Moment Retrieval

  • VCMR extends single-video retrieval to a corpus; XML jointly learns retrieval and localization, while HAMMER uses fine-grained cross-modal interactions and is described as computationally costly.The passage classifies XML as unimodal encoding with late feature fusion, despite its name.Source: §2 · pp. 2–3

Contrastive Learning

  • Contrastive learning distinguishes matching video-query pairs from non-matching pairs, and prior work has applied it to joint video-language representation learning.The related work discusses MI-based methods and MIL-NCE for narrated video-text alignment.Source: §2 · p. 3

3 THE ReLoCLNet MODEL

ReLoCLNet trains separate video and query encoders with contrastive objectives that simulate cross-modal interaction, then uses separate encoding and late fusion for efficient retrieval. Its problem formulation retrieves both a corpus video and a matching temporal moment within that video.

3 THE ReLoCLNet MODEL

  • ReLoCLNet uses VideoCL for video retrieval and FrameCL for moment localization during training, then separately encodes video and query at inference.The model fuses the representations for joint retrieval and localization.Source: §3 · p. 3

3.1 Problem Formulation

  • VCMR jointly requires finding the video containing the query-matching moment and locating that moment within the selected video.The target is a temporal interval represented by start and end positions in the video feature sequence.Source: §3.1 · p. 3

3.2 Query Encoder

The query encoder transforms word features into contextual representations and aggregates them into a modularized query vector. It can produce separate vectors for matching visual and subtitle features when subtitles are available.

3.2 Query Encoder

  • The query encoder projects extracted word features to dimension d, adds positional embeddings, and contextualizes them with transformer blocks.The described transformer block uses multi-head attention and a feed-forward layer, with residual connections and layer normalization.Source: §3.2 · pp. 3–4
  • Additive attention aggregates contextual word representations into a modularized query vector for matching.The encoder uses two transformer blocks before aggregation.Source: §3.2 · p. 4
  • When subtitles are available, the encoder computes separate query vectors for visual and subtitle features; otherwise, it computes only the visual vector.Figure 2 depicts query outputs alongside video representations used for retrieval and moment localization.Source: §3.2 · p. 4, Figure 2 · p. 4

3.3 Video Encoder

The video encoder projects visual and subtitle features into a shared dimension, then contextualizes them with transformer blocks. Without subtitles, the subtitle pathway is removed and only video features are output.

3.3 Video Encoder

  • Visual and subtitle features are projected to dimension d, positionally embedded, and passed through a transformer block.The projected sequences are then processed by co-attentional transformer blocks to capture cross-modal representations within a video.Source: §3.3 · p. 4
  • Standard transformer blocks refine the video and subtitle cross-modal representations separately into final outputs.The outputs are H_v and H_s, respectively.Source: §3.3 · p. 4
  • Without subtitles, the subtitle encoding pathway is removed, co-attention becomes a standard transformer, and only H_v is output.Source: §3.3 · p. 4

3.4 Video Retrieval Module

The video retrieval module compares query vectors with video-frame features using cosine similarity and late feature fusion, then trains retrieval with a hinge loss over negative pairs.

3.4 Video Retrieval Module

  • Each query is represented by visual and, when available, subtitle vectors, while each video is represented by a sequence of corresponding feature vectors.Source: §3.4 · p. 4
  • Cosine similarities between query vectors and video features provide the late-fusion matching scores.The module selects the maximum score across a video's temporal features; when subtitles are available, it averages visual and subtitle scores.Source: §3.4 · pp. 4–5
  • A hinge loss trains video retrieval using sampled non-matching query and video pairs, with a predefined margin Δ = 0.1.Source: §3.4 · p. 5

3.5 Moment Localization Module

The moment localization module uses late feature fusion to score video-query features and predict moment boundaries. It trains boundary prediction against the ground-truth start and end positions.

3.5 Moment Localization Module

  • For efficiency, query and video features are combined through late feature fusion to compute video-query similarity scores.The query representation is first transformed by a feed-forward layer; visual and subtitle scores are averaged when subtitles are available.Source: §3.5 · p. 5
  • A convolutional start-end boundary predictor produces start and end scores, whose joint probability determines the predicted moment boundaries.The selected start and end indices must satisfy 0 ≤ i_s ≤ i_e ≤ n_v − 1.Source: §3.5 · p. 5
  • Cross-entropy training uses one-hot labels for the ground-truth moment's start and end boundaries.Source: §3.5 · p. 5
  • ReLoNet's base architecture comprises query and video encoders plus video retrieval and moment localization modules.The paper next adds contrastive learning objectives to this base model to develop ReLoCLNet.Source: §3.5 · p. 5

3.6 Video and Frame Contrastive Learning

ReLoNet's late-fusion retrieval and localization depend on the separate query and video encoders. ReLoCLNet adds video- and frame-level contrastive objectives to guide those encoders during training.

3.6 Video and Frame Contrastive Learning

  • The base model's retrieval and localization rely on the separately encoded query and video features combined by late feature fusion.Source: §3.6 · p. 5
  • VideoCL and FrameCL guide the encoders to distinguish matching video-query pairs and matching moments from non-matching ones.The objectives are designed for video retrieval and moment localization, respectively.Source: §3.6 · p. 5

3.6.1 Video Contrastive Learning (VideoCL).

  • VideoCL trains video and query encoders to bring matching video-query pairs closer and non-matching pairs farther apart in a joint feature space.It uses NCE over positive matching pairs and sampled negative pairs, with mappings that project both representations into the same embedding space.Source: §3.6.1 · p. 5

3.6.2 Frame Contrastive Learning (FrameCL).

  • FrameCL treats features inside the target moment as foreground and features outside it as background, maximizing query-foreground MI while minimizing query-background MI.It uses a Jensen-Shannon MI estimator because MI estimation for continuous random variables is generally intractable.Source: §3.6.2 · pp. 5–6
  • Both contrastive losses update the video and query encoders, so the objectives mutually affect their representations alongside other losses.Source: §3.6.2 · p. 6

3.7 Training and Inference

ReLoCLNet balances four training-loss components and, at inference, retrieves top-ranked videos before scoring candidate moments and combining video and localization scores.

3.7 Training and Inference

  • λ1 is set to 1.0 and λ2,3,4 to 0.01 to keep the four loss components at the same order of magnitude.The passage describes this as giving equal contributions from the four components.Source: §3.7 · p. 6
  • At inference, the system retrieves the top 100 videos by query–video similarity, then scores candidate moments within each retrieved video.The implementation uses K = 100.Source: §3.7 · p. 6
  • The passage defines a target-moment length quantity as 4𝑛𝑡 = 𝑖𝑒 − 𝑖𝑠 + 1.Source: §3.7 · p. 6
  • The final VCMR score balances video retrieval and moment localization scores using an exponential term and hyperparameter γ.Source: §3.7 · p. 6

4 EXPERIMENTS

Experiments on TVR and ANetCaps find ReLoCLNet's VCMR performance comparable to cross-modal-interaction baselines, while retrieval remains similar in efficiency to XML and much faster than HAMMER. Ablations and qualitative examples examine the distinct contributions of VideoCL and FrameCL.

4.1 Datasets and Evaluation Metrics

  • Experiments use ANetCaps and TVR, evaluate VCMR and its VR and SVMR subtasks, and apply Recall@k with IoU thresholds for moment retrieval.ANetCaps uses I3D visual features and GloVe text embeddings; TVR uses provided features, including RoBERTa text features. Experiments were implemented in PyTorch 1.7.0 with CUDA 11.1 on dual RTX 3090 GPUs.Source: §4.1 · p. 6, §4.2 · p. 7

4.3 Performance Comparison

  • On TVR, ReLoCLNet beats all baselines at Recall@10/100 but trails FLAT and HAMMER at Recall@1; on ANetCaps, it beats them at Recall@1 but trails at Recall@10/100.The authors characterize its overall effectiveness as comparable with FLAT and HAMMER.Source: Table 2 · p. 7, §4.3 · p. 7

4.4.2 Video Retrieval Subtask.

  • On video retrieval, ReLoCLNet outperforms all baselines on both datasets, including a 5.59% improvement in Recall@1 over XML on TVR and a 9.64% score versus HAMMER's 5.89% on ANetCaps.The reported comparisons are for the VR subtask.Source: Table 4 · p. 7, §4.4.2 · p. 8

4.4.3 Single Video Moment Retrieval Subtask.

  • On SVMR, ReLoCLNet leads on TVR and improves substantially over ReLoNet on ANetCaps, but trails FLAT and HAMMER there.The authors relate the latter gap to those models' sophisticated, computationally expensive cross-modal interactions.Source: Table 5 · p. 7, §4.4.3 · p. 8

4.5 Qualitative Analysis

  • On TVR, ReLoCLNet's relative gains over ReLoNet are larger under stricter IoU thresholds for VCMR, SVMR, and video retrieval.For VCMR at IoU=0.5, the reported gains are 47.07% for Recall@1 (8.03 vs 5.46) and 28.35% for Recall@10 (21.37 vs 16.65).Source: Figure 4 · p. 8, Figure 5 · p. 8, Figure 6 · p. 8

4.4.1 Retrieval Efficiency.

  • ReLoCLNet is around 56.71 times faster than HAMMER in retrieval, with efficiency comparable to XML; its contrastive objectives add no inference-time parameters or computation.Retrieval timing excludes data preprocessing and pretrained feature extraction, which are shared across methods.Source: Table 3 · p. 7, §4.4.1 · p. 8

4.5 Qualitative Analysis

  • In two ANetCaps examples, predictions with FrameCL are closer to ground truth than predictions from ReLoNet or ReLoNet with VideoCL.FrameCL is designed to focus on query-matching frames inside the target moment and guide boundary search.Source: Figure 7 · p. 9, §4.5 · p. 9

4.4.4 Analysis on the Learning Objectives.

  • VideoCL improves VCMR and VR, while FrameCL improves all three tasks; combining them further boosts VCMR, VR, and SVMR.The authors describe VideoCL as video-query pair discrimination and FrameCL as distinguishing matching from non-matching moments within a video.Source: Table 6 · p. 8, §4.4.4 · p. 9

5 CONCLUSION

ReLoCLNet combines unimodal encoding with VideoCL and FrameCL to address the efficiency–quality trade-off in VCMR. The paper reports high efficiency and accuracy comparable to more computationally costly state-of-the-art methods.

5 CONCLUSION

  • ReLoCLNet achieves high efficiency while maintaining retrieval accuracy comparable to state-of-the-art methods that require more computation.The conclusion presents unimodal encoding with contrastive learning as a promising direction for VCMR.Source: §5 · p. 9
  • VideoCL brings matching video-query pairs closer and non-matching pairs farther apart at the video level.Source: §5 · p. 9
  • FrameCL operates at frame level to simulate fine-grained cross-modal interaction within a video.Source: §5 · p. 9

Related papers

Loading 2105.06247v1…