PaperlayerView exact revision on arXiv

Source-linked AI summary

Use What You Have: Video Retrieval Using Representations From Collaborative Experts

Yang Liu, Samuel Albanie, Arsha Nagrani, Andrew Zisserman

arXiv:1907.13487v2cs.CV

TL;DR

The paper addresses video retrieval from free-form text queries spanning general and highly specific content. It proposes Collaborative Experts, which combines pretrained multimodal embeddings into a compact joint representation, and reports improved or competitive retrieval across multiple benchmarks while finding specific OCR and ASR cues challenging to use effectively.

  • Problem

    Free-form video queries can contain both general and specific information, making compact video representation learning from high-dimensional sensory data difficult and costly to annotate.

  • Method

    Collaborative Experts combines pretrained embeddings from general and specific video cues through collaborative gating into a single compact video-text embedding for retrieval.

  • Results

    Across MSR-VTT, LSMDC, and MSVD, Collaborative Experts improves or matches prior retrieval methods, while specific OCR and ASR features remain challenging to use effectively.

  • Takeaways & Limitations

    Strong generic pretrained features combined through an appropriate gating mechanism can boost retrieval performance, and the resulting video embeddings can be pre-computed and indexed for efficient retrieval.

  • Takeaways & Limitations

    The paper contains a correction to results reported in its previous version.

Abstract

from arXiv · show

The rapid growth of video on the internet has made searching for video content using natural language queries a significant challenge. Human-generated queries for video datasets `in the wild' vary a lot in terms of degree of specificity, with some queries describing specific details such as the names of famous identities, content from speech, or text available on the screen. Our goal is to condense the multi-modal, extremely high dimensional information from videos into a single, compact video representation for the task of video retrieval using free-form text queries, where the degree of specificity is open-ended. For this we exploit existing knowledge in the form of pre-trained semantic embeddings which include 'general' features such as motion, appearance, and scene features from visual content. We also explore the use of more 'specific' cues from ASR and OCR which are intermittently available for videos and find that these signals remain challenging to use effectively for retrieval. We propose a collaborative experts model to aggregate information from these different pre-trained experts and assess our approach empirically on five retrieval benchmarks: MSR-VTT, LSMDC, MSVD, DiDeMo, and ActivityNet. Code and data can be found at www.robots.ox.ac.uk/~vgg/research/collaborative-experts/. This paper contains a correction to results reported in the previous version.

1 Introduction

The paper targets free-form text retrieval of videos containing rich temporal and multimodal information. It proposes combining pretrained experts, including general and specific cues, into a compact representation for efficient retrieval.

  • Video representation is difficult because videos contain evolving temporal information and multiple modalities, including audio.
  • Free-form queries may describe either general video content or specific details such as speech, identities, and overlaid text.
  • Collaborative Experts combines pretrained embeddings from multiple perception tasks into one compact video representation for joint video-text retrieval.
  • The model investigates generic features alongside OCR and ASR cues, but specific features are intermittently available and challenging to exploit effectively.
  • The combined representation improves retrieval performance on several benchmarks compared with prior work.

2 Related Work

Prior work embeds images, audio, or visual video content with language, but often does not exploit the varied information available across video modalities. Collaborative Experts addresses this broader video-text embedding setting.

  • Earlier cross-modal methods jointly embed images and text, while later work also studies audio-visual and audio-text embeddings.
  • Many video-text retrieval approaches adapt image-text methods and focus primarily on single visual frames.
  • The paper’s setting is more challenging because videos contain high-dimensional information across motion, speech, sounds, and other modalities.
  • The paper notes that its reported finding about specific features differs from the previous version.

3 Collaborative Experts

The collaborative experts framework combines pretrained modality-specific video embeddings into a compact joint representation, using cross-expert attention to retain or suppress information before text–video retrieval.

  • Joint embedding and retrieval: Video and text encoders independently map their inputs into a shared embedding space, allowing retrieval through direct distance comparisons.Paired video and text embeddings are trained to be close, while mismatched pairs are separated; test-time similarity uses an inner product.
  • Video expert representations: Pretrained single-modality experts produce variable-length task-specific embeddings that are temporally aggregated into fixed-size representations.Average pooling handles slower visual features, while NetVLAD aggregates more dynamic audio and word features.
  • Collaborative gating: Collaborative gating predicts attention vectors from pairwise relationships among expert projections, so each expert’s weighting considers the other experts.The pairwise and aggregation functions are implemented as multilayer perceptrons.
  • Collaborative gating: Element-wise sigmoid gating re-calibrates expert activations, dynamically highlighting or suppressing information before the outputs are concatenated.Each modulated expert response passes through a Gated Embedding Module, producing l2-normalized outputs for the final fixed-length video vector.
  • Text query encoder: The text encoder aggregates pretrained contextual word features, projects them into expert-specific subspaces, and assigns normalized mixture weights across projections.The mixture weights are computed from aggregated text features using a linear layer followed by softmax.
  • Missing modalities: Missing experts are zero-padded during similarity estimation, while the remaining mixture weights are renormalized to compensate for changed embedding scale.This handles intermittently unavailable signals such as speech absent from an audio track.

4 Experiments

The experiments evaluate Collaborative Experts across five video-retrieval benchmarks and ablate aggregation methods, expert choices, and caption availability. CE generally improves or matches prior methods, while collaborative gating, diverse experts, and additional captions each contribute to performance.

  • Benchmark evaluation: CE is evaluated on MSR-VTT, LSMDC, MSVD, DiDeMo, and ActivityNet-captions across challenging video domains.The datasets include YouTube videos, personal collections, and movies.
  • Benchmark evaluation: CE consistently improves retrieval performance over existing state of the art on MSR-VTT for both sentence-to-video and video-to-sentence retrieval.The comparison reports recall and rank metrics, with higher recall and lower median or mean rank preferred.
  • Benchmark evaluation: CE matches or outperforms prior work on LSMDC and MSVD, including a prior method using additional COCO images and captions but fewer experts.The LSMDC and MSVD results are reported together in Table 2.
  • Benchmark evaluation: CE outperforms prior work on DiDeMo and is competitive on ActivityNet paragraph-video retrieval under the same video-level annotation setting.The ActivityNet comparison uses the val1 test split.
  • Ablation studies: Collaborative gating improves aggregation performance and yields a more efficient parameterisation than the prior state-of-the-art MoEE method.The aggregation ablation compares CE with expert concatenation, ungated variants, and MoEE using the same experts.
  • Ablation studies: Expert effects vary considerably, stronger features within a modality help, and adding 1 to 20 captions per video improves performance approximately as much as adding the full expert collection.Multiple captions and multiple experts together produce the most robust embedding.
  • Qualitative results: Qualitative retrievals are often reasonable, but the model can rank visually similar videos above the ground truth when signals such as car colour matter.Figure 3 illustrates both reasonable rankings and a failure to differentiate certain signals.

5 Conclusion

The paper concludes that Collaborative Experts combines pretrained features through gating to improve video retrieval, and identifies clustering and summarisation as future applications.

  • Conclusion: Collaborative Experts combines a range of pretrained features through a gating mechanism to boost retrieval performance.The framework learns a joint video-text embedding for efficient retrieval.
  • Conclusion: Future work will explore Collaborative Experts for video clustering and summarisation.

A.1 Paper update, result corrections and summary of differences

The paper corrects an implementation bug that overestimated prior model performance and distorted conclusions about expert importance. The update reruns the experiments, changes hyperparameter selection, adds pretrained experts, and revises the analysis.

  • Correction: An open-source implementation bug caused model performance to be overestimated and the relative importance of experts to be assessed inaccurately.The leak exposed target-video feature availability to the query encoder during training and testing through incorrect embedding-distance weighting.
  • Correction: The correction removes the bug and repeats each experiment reported in the initial paper.
  • Changes: The updated experiments use a systematic approach to hyperparameter selection and include additional pretrained expert features.These additions assess feature strength within a modality and accompany corresponding updates to the written analysis.

A.2 Detailed Description of Datasets

The evaluation spans five video-retrieval benchmarks covering diverse videos, captions, movie clips, paragraph descriptions, and personal videos. Dataset sizes, splits, and modality availability differ across benchmarks.

  • MSR-VTT: MSR-VTT contains approximately 200K video-caption pairs from 10K YouTube clips and uses 6513, 497, and 2990 videos for training, validation, and testing.The dataset offers broad video diversity but includes reasonably high label noise from duplicate annotations.
  • MSVD: MSVD uses a standard split of 1,200, 100, and 670 videos for training, validation, and testing, and its videos do not have audio streams.
  • LSMDC: LSMDC contains 118,081 short clips extracted from 202 movies, with captions from scripts or descriptive video services.Its validation set contains 7408 clips, and evaluation uses a 1000-video test set.
  • ActivityNet and DiDeMo: ActivityNet Captions contains 20K YouTube videos paired with approximately 100K descriptive sentences and evaluates paragraph-video retrieval on val1.The val1 split allocates 10,009 videos for training and 4,917 for testing.
  • ActivityNet and DiDeMo: DiDeMo contains 10,464 unedited personal videos in diverse visual settings with roughly 3-5 description pairs per video.

A.3 Optimisation details and hyperparameter selection

Hyperparameters are selected through dataset-specific searches, beginning with single-expert grid searches and followed by a projection-dimension comparison on MSR-VTT. ActivityNet instead reuses the DiDeMo configuration because its validation set is used for assessment.

  • Hyperparameter search: Each dataset receives a grid search over batch size, learning rate, and weight decay using a single expert to determine optimisation parameters.The searched batch sizes are 16, 32, 64, 128, and 256; learning rates are 0.1 and 0.01; weight decays are 1E-3 and 5E-5.
  • Projection dimension: An MSR-VTT experiment compares projection dimensions of 512, 768, and 1024 features after the initial optimisation search.
  • Dataset-specific settings: ActivityNet hyperparameters are copied from the DiDeMo configuration because its validation set is used to assess performance.Hyperband searches use a budget of 500 sampled configurations.
  • Reproducibility: The updated implementation makes the experiment configurations, experts, pretrained models, and logs available.

A.4 Ablation Studies - Full Tables

The appendix provides full ablation-study tables examining expert importance and training-caption choices. The supplied passages identify the table topics but do not report their numerical findings.

  • Expert ablations: Table 7 studies the importance of each expert when combined with Scene features.
  • Expert ablations: Table 8 studies the importance of experts on the MSR-VTT dataset.
  • Caption ablations: Table 9 studies how the number of captions used in MSR-VTT training affects the ablation results.

A.5 Implementation Details

The implementation extracts pretrained visual, action, face, audio, scene, and text features using modality-specific preprocessing and aggregation. Model capacity is also examined by varying the shared embedding dimension on MSR-VTT.

  • Capacity ablation: Table 10 varies CE’s shared embedding dimension to study model capacity on MSR-VTT.The supplied caption identifies the ablation’s dataset and purpose but does not provide the tested dimensions or results.
  • Action and face features: Action features combine I3D and R(2+1)D models with temporal windows or short clips sampled from video frames.Action(KN) uses 64-frame windows with stride 25, while Action(IG) processes eight consecutive 112 × 112 pixel frames at 30 fps.
  • Audio and scene features: Audio is resampled to 16kHz mono, converted into log mel-spectrograms, and mapped into 128-dimensional vectors from 0.96-second frame collections.Scene embeddings are extracted from 224 × 224 pixel centre crops sampled at 1 fps using a DenseNet-161 model.
  • Text features: Text words are encoded with Google News word2vec, contextualized by pretrained OpenAI-GPT, and aggregated sentence-wise using NetVLAD.The word2vec resource is identified as GoogleNews-vectors-negative300.bin.gz.
Loading 1907.13487v2…