PaperlayerView exact revision on arXiv

Source-linked AI summary

MDMMT: Multidomain Multimodal Transformer for Video Retrieval

Maksim Dzabraev, Maksim Kalashnikov, Stepan Komkov, Aleksandr Petiushko

arXiv:2103.10699v1cs.CV

TL;DR

Text-to-video retrieval needs broad coverage beyond individual datasets and reliable benchmark separation. The paper combines multiple video caption datasets for a multidomain retrieval model, achieving state-of-the-art results on MSRVTT and LSMDC and across three benchmarks without finetuning, while analyzing dataset overlap.

  • Problem

    Individual video datasets do not cover a significant portion of real-life situations, motivating broader training data for video retrieval.

  • Method

    The paper combines multiple video caption datasets to train a multidomain text-to-video retrieval model and uses an overlap-detection approach for dataset analysis.

  • Results

    The model achieves state-of-the-art results on MSRVTT and LSMDC and shows the best results on several datasets simultaneously without finetuning.

  • Takeaways & Limitations

    Training on multiple datasets can support multidomain generalization, while train-test overlap can materially affect benchmark performance.

  • Takeaways & Limitations

    The authors do not know whether Kinetics–MSRVTT overlap affects overfitting and cannot assess overlap involving non-public IG65M and CLIP datasets.

Abstract

from arXiv · show

We present a new state-of-the-art on the text to video retrieval task on MSRVTT and LSMDC benchmarks where our model outperforms all previous solutions by a large margin. Moreover, state-of-the-art results are achieved with a single model on two datasets without finetuning. This multidomain generalisation is achieved by a proper combination of different video caption datasets. We show that training on different datasets can improve test results of each other. Additionally we check intersection between many popular datasets and found that MSRVTT has a significant overlap between the test and the train parts, and the same situation is observed for ActivityNet.

1 Introduction

Text-to-video retrieval seeks relevant video segments for natural-language queries, but effective systems need broad coverage of real-world situations. The introduction motivates combining datasets and contrasts scalable two-stream retrieval with less-scalable single-stream fusion.

  • Task motivation: 500+ hours of video are uploaded to YouTube every minute, increasing the importance of modern video search methods.The paper also connects this need to the growing amount of video stored on mobile devices.
  • Task motivation: Text-to-video retrieval returns relevant video segments for a natural-language description of desired content.Queries may describe objects, actions, sounds, and relations.
  • Fusion approaches: Single-stream approaches fuse query and video from the beginning but require a full forward pass for every query-video pair.They can access all input data early, yet this computation is not scalable for gallery search.
  • Fusion approaches: Two-stream networks separately encode text and video into a shared embedding space, enabling precomputed video embeddings for scalable retrieval.A text query is then compared with video representations rather than recomputing every video representation for each query.
  • Dataset coverage: No existing video dataset covers a significant portion of real-life situations, motivating rules for combining datasets into one larger training database.The authors frame dataset combination as an early step toward broader situation, action, and object coverage.
  • Contributions: The paper contributes state-of-the-art results on MSRVTT and LSMDC, strong performance across three benchmarks without finetuning, and a dataset-overlap analysis approach.These contributions combine model performance claims with a practical method for identifying train-test overlap.

2 Related work

Related work covers benchmark construction and evaluation issues, contrastive objectives, and multimodal temporal aggregation for video retrieval. It also describes MMT as a two-stream architecture using pretrained modality experts, BERT text features, and weighted similarity.

  • Datasets: MSRVTT contains 10k video segments with 20 captions each, while ActivityNet contains 20k videos and 100k captions covering most full videos.These datasets are central examples of caption-based resources used for retrieval evaluation and training.
  • Datasets: MSRVTT’s official full split separates 7k training videos from 3k test videos and avoids shared source videos or queries across the split.The listed properties define the intended separation of the official split.
  • Evaluation caveats: The 1k-A and 1k-B MSRVTT splits mix train and test content, violating separation properties satisfied by the full split.Both alternative splits sample different 1k-video test sets and use only one caption per segment.
  • Evaluation caveats: All described MSRVTT splits contain train-test overlap, so the authors create a clean split by removing overlapping videos only from training.The test portion remains unchanged in the clean split.
  • Evaluation caveats: ActivityNet paragraph retrieval concatenates captions into long queries and retrieves whole videos, making it distinct from classical video retrieval and enlarging model inputs.The authors instead sample one random segment from each validation video for their video-retrieval evaluation.
  • Training objectives: Contrastive learning brings matched video-text pairs together while separating mismatched pairs, using bi-directional max-margin ranking or MIL NCE for noisy multiple matches.The ranking loss uses batch size, pairwise similarity, and a predefined margin; MIL NCE reduces distance to several known matching texts.
  • Multimodal architectures: MMT aggregates temporal video tokens with a transformer and processes multiple pretrained modalities, while BERT encodes text and modality weights produce a weighted sum of dot products.The described modalities include motion, RGB, scene, face, OCR, speech, and audio.

3 Methodology

The methodology combines pretrained multimodal retrieval components with multiple video-caption datasets to build one model across domains. It addresses dataset sampling, information size, and train–test overlap when extending the training database.

  • Model design: The model uses pretrained experts for video features, with motion treated as especially important because it captures actions and objects across frame segments.The authors argue that missing information from the motion expert is likely to remain unavailable to the retrieval model.
  • Research questions: The study evaluates motion experts and investigates how to combine several video-caption datasets without specializing the model to one dataset.The stated questions include selecting a pretrained motion expert and preventing train–test overlap when datasets are combined.
  • Multidomain training: The authors choose a single multidomain model rather than separate models specialized for MSRVTT, ActivityNet, and LSMDC.Earlier specialized models perform well on their training domain but poorly on another domain, whereas this work targets all domains simultaneously.
  • Dataset construction: Dataset information size is estimated by the number of training examples needed before performance saturates as training time increases.Figure 2 represents larger information sizes with larger balls, which indicate greater diversity in the data.
  • Dataset sampling: Sampling rules account for dataset size and information differences, while combined training may require more time and a deeper model than training on one dataset.The authors report that carefully extending training with another dataset often improves results for a specific test dataset.
  • Dataset construction: The combined training database includes MSRVTT, ActivityNet, LSMDC, TwitterVines, YouCook2, MSVD, TGIF, and SomethingV2.The combined collection increases video segments by 40 times and unique captions by 4 times compared with MSRVTT.

4 Experiments

The experiments examine motion experts, dataset combination, and final multidomain retrieval performance. Results show that expert pretraining and dataset composition strongly affect performance, while one model achieves strong results across multiple benchmarks.

  • Stronger motion experts: Stronger motion experts usually improve retrieval, but r(2+1)d 152 performs poorly despite strong Kinetics 400 results, whereas r(2+1)d 34 performs much better.The authors suggest r(2+1)d 152 may be overspecialized for Kinetics 400.
  • Stronger motion experts: CLIP and models trained on IG65M outperform supervised Kinetics models, with CLIP ViT-B/32 outperforming the other tested models by a large margin.The comparison includes models trained unsupervised or weakly supervised on CLIP, IG65M, Sports1M, and HowTo100M.
  • Combining datasets: Proper dataset combination usually outperforms training on a single dataset and enables one model to capture knowledge from all used datasets.Training samples datasets by manually adjusted weights, with sampling probability proportional to each dataset’s weight.
  • Final results: The experiments use MMT with motion embeddings from irCSN152 pretrained on IG65M, while final models combine audio, motion, and RGB modalities.The final multimodal configuration uses VGGish for audio, CLIP ViT-B/32 for RGB, and irCSN152 for motion.
  • Final results: 8.7%, 10.5%, and 14.4% R@5 gains over the previous MSRVTT state of the art are reported on the full, 1k-A, and 1k-B splits, while LSMDC improves by 8.6%.The MSRVTT model uses CLIP, irCSN152, and audio, and the LSMDC result is reported against the previous state of the art.
  • Final results: 22.6% improvement over MMT is reported on the authors’ ActivityNet text-to-video split, rather than the standard paragraph-retrieval protocol.The authors also report that tested pooling and caption-length variants do not improve the results.

5 Conclusions and Discussion

The model achieves state-of-the-art text-to-video retrieval results on MSRVTT and LSMDC, while also outperforming the previous state of the art on ActivityNet video retrieval without finetuning. The study additionally identifies dataset overlap that affects evaluation and reports implementation choices for video preprocessing.

  • The model achieves state-of-the-art results on MSRVTT and LSMDC benchmarks.
  • It outperforms the previous state-of-the-art model, MMT, by a large margin on ActivityNet video retrieval.
  • Training on many video caption datasets enables strong results on several datasets simultaneously without finetuning.
  • Significant overlap exists between MSRVTT test and train parts and between ActivityNet test and train parts.
  • The implementation computes one video embedding per second using one-second windows with one-second shifts.

B Datasets combination

Combining datasets improves retrieval performance across MSRVTT, ActivityNet, and LSMDC relative to single-dataset baselines. The full dataset combination produces gains on all three benchmarks.

  • The Mc, A, and L models are single-dataset baselines trained on MSRVTT, ActivityNet, and LSMDC, respectively.
  • +5.5% for MSRVTT, +1.47% for ActivityNet, and +2.74% for LSMDC result from combining all datasets.
  • The Mc→McALVYMT transitions improve MSRVTT, ActivityNet, and LSMDC by +4.85%, +1.45%, and +2.63%, respectively.

C Test and train intersection

The paper develops an overlap-detection procedure for identifying duplicated or similar video segments across training and test datasets. It finds substantial overlap involving MSRVTT and ActivityNet and uses dataset-specific estimates for larger corpora.

  • The method searches overlaps between training datasets and the MSRVTT, ActivityNet, and LSMDC test parts, then removes found duplicates from training.
  • The analysis covers only MSRVTT, ActivityNet, and LSMDC test parts because the authors lacked sufficient human resources to examine all other datasets.
  • The procedure compares video segments using frame embeddings, interval-level cosine similarity, and manual assessment of highest-scoring candidate pairs.
  • Significant overlap occurs between MSRVTT test and train parts, including the full, 1k-A, and 1k-B splits, and ActivityNet has the same problem.
  • HowTo100M may contain about 300 video segments, or 10% of the MSRVTT full test part, overlapping with it.
  • Kinetics700 may contain approximately 500–600 video segments, or 10% of the ActivityNet test set, duplicated in ActivityNet validation.

C.1.2 Number of pairs to assess

The paper estimates how many duplicate pairs remain after reviewing the highest-scoring candidates. It calibrates this estimate with augmented positives and negative pairs to construct a search curve.

  • Augmented query videos are created by random 70%–100% side cropping and a random 0–1 second temporal shift.
  • The total number of duplicate pairs is estimated as M/F^-1(N), where N+M pairs were examined and M duplicates were found.
  • The search curve F(x) relates the fraction of found duplicate pairs to the number of negative pairs requiring assessment.
  • Positive scores compare each query video with its augmented version, while negative scores compare query videos with gallery videos.

C.1.3 Best 2D feature extractor

The duplicate-search system compares pretrained feature extractors using search curves that estimate manual inspection effort. ResNeXt-101-32x48d-WSl performs best on the fixed benchmark and is selected for duplicate searches, while the curve must be re-estimated for each dataset pair.

  • Search-curve evaluation: The benchmark compares feature extractors on MSRVTT full train against 596k random HowTo100M videos.Each HowTo100M video contributes a random 30-second segment for comparison.
  • Search-curve evaluation: Search curve F estimates how many negative pairs assessors must inspect to find a target fraction of positive pairs.Lower curves are better because they require fewer false candidates to be inspected manually.
  • Feature-extractor selection: ResNeXt-101-32x48d-WSl achieves the best search-curve result and is used for duplicate searching.The comparison includes ImageNet-, Places365-, and Instagram-pretrained networks.
  • Scope: The search curve depends significantly on the data, so it should be estimated separately for every query-gallery dataset pair.The reported extractor choice comes from a fixed benchmark and may not generalize unchanged across dataset pairs.
  • Preprocessing: Screensavers and near-black frames can dominate duplicate candidates despite different remaining video content.The system therefore suppresses embeddings associated with detected screensavers or frames dominated by one color.

C.2 Cleaning results

Cleaning removes duplicate training examples identified through YouTube IDs and manual assessment, and substantially lowers measured performance on affected MSRVTT splits. The reported cleaning comparison focuses on original MMT results.

  • Cleaning procedure: The cleaning procedure first removes training segments sharing YouTube IDs with MSRVTT or ActivityNet test parts, then manually assesses high-scoring embedding matches.These are the two stages used for the final duplicate-cleaning result.
  • Cleaning findings: More than 100 duplicate pairs are found within both MSRVTT and ActivityNet, making them the most problematic datasets for overlap.HowTo100M and Kinetics700 are also difficult to assess fully because of their large size.
  • Scope: The 1k-A and 1k-B manual cleaning is incomplete because cleaning was performed only for the full split.Moving videos between test and train in these splits introduces additional overlap that was not removed.
  • Performance impact: After cleaning, original MMT performance decreases significantly on the MSRVTT 1k-A and 1k-B splits.The reported comparison includes no cleaning, YouTube-ID cleaning, and combined YouTube-ID plus manual cleaning.

C.2.1 Intersection by YouTube ID and embeddings

The intersection analysis combines YouTube-ID matching with embedding-based search and manual assessment to estimate cross-dataset duplicates. It identifies substantial overlap in several dataset pairs but can require impractical manual effort for full estimation.

  • Two-stage intersection analysis: The first intersection stage matches video segments by YouTube ID, while the second searches embeddings and manually assesses high-scoring pairs.Table 9 reports the final result after applying both stages.
  • YouTube-ID intersection: MSRVTT 1k-A and 1k-B each show about 38% YouTube-ID overlap between their corresponding train and test parts.The original MSRVTT full split has no YouTube-ID overlap between train and test parts.
  • Estimation method: The estimated total duplicate count uses search curve F and scales observed duplicates from the manually assessed pairs.For a HowTo100M–MSRVTT example, 5k assessed pairs with 15 duplicates imply approximately 320 total duplicates and 106k pairs for full assessment.
  • Practical boundary: Full intersection estimation can be impractical when the extrapolated number of pairs requiring manual assessment is too large.The HowTo100M–MSRVTT example requires an estimated 106k manual assessments for full overlap discovery.

E Pretrained model

The pretrained-model evaluation tests the same multimodal models across datasets without target-specific finetuning. Pretraining improves MSRVTT and ActivityNet retrieval, while LSMDC performance remains approximately unchanged.

  • Evaluation setup: The models are evaluated on MSRVTT, ActivityNet, and LSMDC without special finetuning for the target dataset.The comparison includes models trained with and without HowTo100M pretraining.
  • MSRVTT: HowTo100M pretraining increases MSRVTT R1 by 1% and R5 by 2%.These gains are reported for the pretrained-model comparison on the MSRVTT full clean split.
  • ActivityNet: HowTo100M pretraining improves ActivityNet R1 by about 2% and R5 by about 4%.The ActivityNet results use the authors’ own subset of the original ActivityNet test part.
  • LSMDC: LSMDC shows approximately the same results with and without HowTo100M pretraining.The corresponding comparison is reported in the LSMDC evaluation table.
Loading 2103.10699v1…