PaperlayerView exact revision on arXiv

Source-linked AI summary

On Semantic Similarity in Video Retrieval

Michael Wray, Hazel Doughty, Dima Damen

arXiv:2103.10095v1cs.CV

TL;DR

Instance-based video retrieval treats only the paired caption and video as relevant, even though other captions may also describe a video. The paper proposes continuous semantic-similarity retrieval and caption-based proxies; its examples and analyses show that instance-based assumptions can overlook relevant matches.

  • Problem

    Instance-based retrieval treats only the paired video-caption item as relevant, although multiple captions may validly describe a video and other matches may be relevant.

    Source: §1 · p. 1, §3 · p. 2

  • Method

    The paper defines continuous cross-modal similarity so multiple items can be equally relevant, and uses caption-to-caption proxies to estimate it without additional annotations.

    Source: §4.1 · p. 4, §4.4 · p. 5

  • Results

    Examples and experiments expose shortcomings of instance-based relevance: captions can be fully or partly relevant, and even high proxy thresholds yield more than one relevant caption on average.

    Source: §5.1 · p. 7, §6 · p. 8

  • Takeaways & Limitations

    Semantic Similarity Video Retrieval represents multiple relevant items and graded similarity rather than limiting relevance to one paired item.

    Source: §4.1 · p. 4, §6 · p. 8

    Direct word matching ignores context and treats synonyms as irrelevant, while BERT and other embedding measures do not provide useful similarity scores on these video-retrieval datasets.

    Source: §4.4 · p. 5

Abstract

from arXiv · show

Current video retrieval efforts all found their evaluation on an instance-based assumption, that only a single caption is relevant to a query video and vice versa. We demonstrate that this assumption results in performance comparisons often not indicative of models' retrieval capabilities. We propose a move to semantic similarity video retrieval, where (i) multiple videos/captions can be deemed equally relevant, and their relative ranking does not affect a method's reported performance and (ii) retrieved videos/captions are ranked by their similarity to a query. We propose several proxies to estimate semantic similarities in large-scale retrieval datasets, without additional annotations. Our analysis is performed on three commonly used video retrieval datasets (MSR-VTT, YouCook2 and EPIC-KITCHENS).

1. Introduction

Video retrieval benchmarks assume that only the caption paired with a video is relevant, although other videos may be equally relevant. The paper proposes semantic-similarity retrieval for evaluation and training, with multiple relevant items ranked by similarity.

1. Introduction

  • Under semantic similarity, multiple videos or captions can be relevant, and their relative order does not affect evaluation; items are ranked by similarity.This contrasts with treating only the collected video-caption pair as relevant.Source: §1 · p. 1
  • The paper challenges instance-based benchmarks and proposes semantic-similarity video retrieval for both evaluation and training.Its analysis uses caption-to-caption proxies to estimate semantic similarity and examines their impact on three benchmark datasets.Source: §1 · p. 1
  • The paper analyzes MSR-VTT, YouCook2, and EPIC-KITCHENS using caption-to-caption proxies that require no additional annotations.The analysis examines proxy effects on current baselines and evaluations.Source: §1 · p. 1

2. Related Work

Prior work on semantic retrieval includes image methods and video methods that mostly retain instance-based evaluation. This paper positions semantic-similarity retrieval as an alternative for video retrieval evaluation and training.

2. Related Work

  • Image-retrieval work used semantic annotations, learned semantic spaces, or caption-derived proxies, but many approaches retained binary relevance.Gordo and Larlus used caption-based semantic proxies; Chun et al. incorporated plausible matches through class knowledge while retaining binary relevance.Source: §2.1 · p. 2

2.1. Semantic Image Retrieval

  • Kim et al. [30] explored non-binary relevance in image retrieval using a log-ratio loss and pose distances, and investigated word mover’s distance as a semantic proxy.The passage describes this as prior work addressing relevance beyond binary and instance-based labels in both training and evaluation.Source: §2.1 · p. 2

2.2. Video Retrieval

  • Video-retrieval methods largely train and evaluate instance-based retrieval, while prior semantic-similarity training approaches still use instance-based evaluation.The paper identifies its work as the first to scrutinize instance-based correspondence in video-retrieval benchmarks and proposes semantic similarity for evaluation and training.Source: §2.2 · p. 2

3. Shortcomings of Current Video Retrieval Benchmarks

The section argues that instance-based video-retrieval benchmarks overlook valid and partially relevant matches because captions were collected with individual videos. Its examples show why paired-caption relevance can misrepresent retrieval quality.

3. Shortcomings of Current Video Retrieval Benchmarks

  • Instance-based retrieval assumes only the collected video-caption pair is relevant, although captions were collected while annotators observed a single video.The section notes that this collection process could not account for a caption’s relevance to other videos.Source: §3 · pp. 2–3
  • Figure 2 illustrates that captions deemed irrelevant by benchmarks can be highly relevant, and the top MSR-VTT example is more specific than its paired ground-truth caption.The figure presents test-set examples from three datasets.Source: Figure 2 · p. 3, §3 · p. 3
  • The analysis focuses on MSR-VTT, YouCook2, and EPIC-KITCHENS, with the latter two selected for their shared cooking scenario.The section says a shared scenario increases the number of relevant captions within a dataset.Source: §3 · p. 3
  • Different captions can all validly describe a video, so methods that retrieve them in a different order may be unfairly penalized by paired-caption ground truth.The captions can vary in synonyms, sentence structure, or descriptive detail.Source: §3 · p. 3
  • Some captions are only partly interchangeable yet remain more relevant than unrelated captions, such as descriptions of opening other bottles rather than cutting a tomato.The passage gives an EPIC-KITCHENS example contrasting bottle captions with an irrelevant action.Source: §3 · p. 3
  • Because manually annotating all relevant and partly relevant captions is impractical in time and cost, the paper proposes similarity proxies requiring no extra annotation.The proxies use external corpora or knowledge bases.Source: §3 · p. 3

4. Video Retrieval with Semantic Similarity

The paper replaces single-match instance-based retrieval with semantic similarity for video-to-text and text-to-video retrieval. It defines continuous relevance, proposes nDCG evaluation and similarity-based training, and estimates relevance using caption-derived proxies.

4. Video Retrieval with Semantic Similarity

  • The framework applies to both video-to-text and text-to-video retrieval, ranking items from the opposing modality by distance in the learned embedding space.Overall evaluation considers both retrieval directions.Source: §4 · p. 3, §4.2 · p. 4

4.1. Definition

  • Semantic similarity assigns each video-caption pair a continuous score from 0 to 1, allowing multiple equally relevant matches and graded relevance.Retrieval order among items with the same score should not affect evaluation.Source: §4.1 · p. 4

4.2. Evaluation

  • nDCG evaluates retrieval using multiple relevant items and their graded similarities, avoiding the single-item or binary-relevance constraints of common instance-based metrics.It treats items with equal similarity as equally relevant regardless of retrieval order.Source: §4.2 · p. 4

4.3. Training

  • Training can use triplets whose positive and negative pairs fall above and below a chosen semantic-similarity threshold, respectively.The authors report that log-ratio loss performed worse than thresholding the triplet loss; adapting alternative losses remains future work.Source: §4.3 · p. 4

4.4. Proxy Measures for Semantic Similarity

  • Caption-derived proxies estimate semantic similarity without extra annotation, using bag-of-words, part-of-speech, synset-aware, or METEOR measures.BoW ignores word context and treats synonyms as unrelated, while synsets account for synonyms and METEOR also considers word order; embedding-model proxies were not useful on these datasets.Source: §4.4 · p. 5

5. Semantic Similarity Analysis

The section compares four semantic-similarity proxies across three retrieval datasets, examines distortions in instance-based evaluation, and tests proxy-based evaluation and semantic training.

5. Semantic Similarity Analysis

  • The analysis compares four proxies across three datasets and asks how they differ, affect IVR evaluation, perform under SVR, and respond to semantic training.The qualitative examples show that proxy scores differ while methods agree on highly relevant and irrelevant captions.Source: §5 · pp. 5–6

5.1. Proxy Measure Comparisons

  • Proxy choice changes what counts as similar: BoW requires exact word sets, SYN links verb/noun synonyms, and METEOR rewards word-order alignment.Average relevant-caption counts remain above one even at high thresholds; SYN includes as many or more than PoS, especially on EPIC-KITCHENS.Source: §5.1 · pp. 6–7, Figure 4 · p. 7

5.2. Shortcomings of IVR Evaluation

  • Synset-aware relevance changes Geometric Mean substantially (~30 EPIC-KITCHENS, ~6.0 MSR-VTT, ~5.0 YouCook2), exposing limits of instance-based ranking.The reported-to-upper-bound gap indicates that baselines sometimes retrieve highly similar captions above the ground-truth caption.Source: §5.2 · p. 7, Figure 5 · p. 7

5.3. Using Semantic Proxies for Evaluation

  • Under semantic-proxy evaluation, MoEE, CE, and JPoSE are comparable to MME in almost all cases, despite outperforming it on instance-based Geometric Mean.MME even outperforms more complex approaches on YouCook2.Source: §5.3 · p. 7, Figure 7 · p. 8

5.4. Training with Semantic Similarity

  • For CE on YouCook2, semantic training improves nDCG; peaks occur near T=0.4 for SYN, METEOR, and PoS and T=0.2 for BoW, with best performance under matched proxies.nDCG values cannot be compared across plots because they use different evaluation proxies.Source: §5.4 · p. 8, Figure 6 · p. 8

6. Conclusion

The paper argues that instance-based benchmarks miss the non-binary similarities between videos and captions, and proposes Semantic Similarity Video Retrieval (SVR). It introduces annotation-free proxies and reports benefits from incorporating semantic knowledge during training.

6. Conclusion

  • SVR allows multiple captions to be relevant to a video and vice versa, representing similarity between items as non-binary.The paper presents this as an alternative to instance-based retrieval.Source: §6 · p. 8
  • Four semantic-similarity proxies require no additional annotations and scale with dataset size.The paper evaluates them on three datasets and provides a public SVR benchmark for those datasets.Source: §6 · p. 8
  • Incorporating semantic knowledge during training can greatly benefit model performance.Source: §6 · p. 8

Supplementary Material

The supplementary material reports a human perceptual study, proxy correlations, additional proxy measures deemed unsuitable for the datasets, and numerical results corresponding to Figure 7.

Supplementary Material

  • The supplement covers human agreement, proxy correlations, unsuitable additional proxies, and a numerical version of Figure 7.The numerical results are provided for comparison in future work.Source: Supplementary Material · p. 11

A. Proxy Measures Human Agreement Study

The study tests how well proxy rankings correspond to human judgments using selected captions for videos from YouCook2 and MSR-VTT. The reported results show good agreement overall, with MET lower than the other proxies.

A. Proxy Measures Human Agreement Study

B. Correlation Between Semantic Proxies

The supplementary analysis finds consistent positive correlations among the semantic proxies across three datasets, with stronger agreement for low-ranked captions. It also defines comparisons with learned text and visual feature proxies.

B. Correlation Between Semantic Proxies

C.1. Definition

  • The additional proxies use cosine similarity for learned representations: Word2Vec and BERT for text, and pretrained video features for visual similarity.The visual proxy assumes video features describe the caption and distinguish videos; Word Mover’s Distance performed slightly worse.Source: §C.1 · pp. 11–12
  • Figure 9 plots the average number of relevant captions against the threshold for each dataset and proxy measure.The caption names Word2Vec, BERT, and Visual (VIS) among the measures shown.Source: Figure 9 · p. 12

C.2. Proxy Measure Comparisons

Learned and pretrained-model proxies can mark many captions as relevant without capturing semantic similarity, making them unsuitable for estimating it in these comparisons.

C.2. Proxy Measure Comparisons

  • Learned proxies yield many more relevant captions than proposed proxies at almost all thresholds, inflating nDCG scores.The comparison covers YouCook2, MSR-VTT, and EPIC-KITCHENS.Source: §C.2 · p. 12
  • Visual similarity is high between segments from the same video, so pretrained visual similarity is unsuitable as a semantic-similarity proxy.Source: §C.2 · p. 12
  • BERT and Word2Vec produce significantly more relevant captions than proposed metrics, partly matching captions by context rather than semantic relevance.For example, “open” and “close” often occur in similar contexts but represent opposite actions, yet both models assign them higher similarity.Source: §C.2 · p. 12

D. Table of Figure 7

Table 3 presents baseline evaluations across three datasets and proxy measures; the main-text discussion is in Section 5.3.

D. Table of Figure 7

Loading 2103.10095v1…