PaperlayerView exact revision on arXiv

Source-linked AI summary

TEACHTEXT: CrossModal Generalized Distillation for Text-Video Retrieval

Ioana Croitoru, Simion-Vlad Bogolin, Marius Leordeanu, Hailin Jin, Andrew Zisserman, Samuel Albanie, Yang Liu

arXiv:2104.08271v2cs.CV

TL;DR

Text-video retrieval has under-explored the use of large-scale language pretraining despite progress from visual and audio pretraining. TeachText distills complementary information from multiple text encoders into a retrieval student, achieving state-of-the-art results on six benchmarks and supporting dataset denoising without additional inference cost.

  • Problem

    Text-video retrieval has under-explored algorithms for exploiting large-scale language pretraining and the complementary information across text embeddings.

  • Method

    TEACHTEXT trains teacher retrieval models with different text embeddings and distills their text-video similarity matrices into a student model.

  • Results

    TEACHTEXT achieves state-of-the-art results on six text-video retrieval benchmarks and adds no computational cost during inference.

  • Takeaways & Limitations

    The method provides a way to exploit complementary text-encoder information and can also be used to denoise text-video retrieval datasets.

  • Takeaways & Limitations

    The method assumes the embeddings are produced by models trained on relevant tasks such as action recognition or language modelling.

Abstract

from arXiv · show

In recent years, considerable progress on the task of text-video retrieval has been achieved by leveraging large-scale pretraining on visual and audio datasets to construct powerful video encoders. By contrast, despite the natural symmetry, the design of effective algorithms for exploiting large-scale language pretraining remains under-explored. In this work, we are the first to investigate the design of such algorithms and propose a novel generalized distillation method, TeachText, which leverages complementary cues from multiple text encoders to provide an enhanced supervisory signal to the retrieval model. Moreover, we extend our method to video side modalities and show that we can effectively reduce the number of used modalities at test time without compromising performance. Our approach advances the state of the art on several video retrieval benchmarks by a significant margin and adds no computational overhead at test time. Last but not least, we show an effective application of our method for eliminating noise from retrieval datasets. Code and data can be found at https://www.robots.ox.ac.uk/~vgg/research/teachtext/.

1. Introduction

Text-video retrieval has benefited from multimodal video encoders, but exploiting diverse language pretraining remains under-explored. TEACHTEXT distills multiple text encoders into a retrieval student, improving performance without inference overhead and reaching state-of-the-art results on six benchmarks.

  • 1. Introduction: TEACHTEXT addresses the under-explored use of multiple language-pretrained text encoders for text-video retrieval.The method is motivated by inconsistent performance across text embeddings and leverages their complementary information.
  • 1. Introduction: The method trains a student from teacher retrieval models using different text embeddings and distills their text-video similarity matrices.This provides an enhanced supervisory signal while preserving a single student model at inference.
  • 1. Introduction: TEACHTEXT complements gains from adding video modalities while imposing no additional computational cost during inference.Unlike adding video modalities, the multiple-text-encoder strategy does not increase test-time computation.
  • 1. Introduction: The approach achieves state-of-the-art performance on six text-video retrieval benchmarks and is also applied to dataset denoising.The denoising application targets noise in modern retrieval datasets.

2. Related Work

Prior text-video retrieval methods learn joint representations of language and video, while related work studies text embeddings and knowledge distillation. These lines of work provide the context for transferring information from multiple text encoders into retrieval models.

  • 2. Related Work: Text-video retrieval commonly uses cross-modal methods that place text queries and video content in a joint-embedding space.The task focuses on retrieving video matching a natural-language description.
  • 2. Related Work: Text embeddings vary in pretraining data, architecture, task adaptation, and model capacity, with prior work showing benefits from finetuning and larger models.Vision-language applications have also motivated embeddings designed specifically for visual tasks.
  • 2. Related Work: Knowledge distillation transfers information from a teacher to a student and has been generalized to learning with privileged information.The related framework includes similarity control and knowledge transfer.

3. Motivation and intuition

The motivation study finds that text embeddings are sensitive to datasets and task conditions, with substantial performance variation and complementary retrieval behavior. These observations support combining multiple embeddings rather than relying on one language representation.

  • 3. Motivation and intuition: Text embeddings show significant performance variance within and across retrieval datasets, indicating that different embeddings capture different information.Their rankings are not consistent across datasets, suggesting no single embedding is uniformly best.
  • 3. Motivation and intuition: The embeddings differ in training sources and design, including vision-oriented word embeddings, book-trained language models, and models trained on crawled data.These differences provide plausible sources of the observed variation in retrieval behavior.
  • 3. Motivation and intuition: Finetuning gpt2-xl on target retrieval queries improves performance, while the embedding still exhibits a domain gap to each corpus.The finetuned version is termed gpt2-xl-F.
  • 3. Motivation and intuition: On MSR-VTT, only around 19% of queries at R1 and 42% at R5 are correctly retrieved by all three examined embeddings.The embeddings are gpt2-xl, gpt2-xl-F, and w2v.

4. Method

TEACHTEXT trains a student retrieval model to match an aggregate of teacher similarity matrices derived from different text embeddings, while retaining the standard retrieval objective. The method transfers cross-modal correspondence information without adding parameters or modalities to the final model, and its matrix-based distillation is architecture-agnostic.

  • Learning the similarity matrix: TEACHTEXT trains a student to match an aggregate of teacher similarity matrices while jointly optimizing retrieval and distillation losses.Teachers use different text embeddings, whereas the student uses a single text embedding; all models share the same video embeddings during training.
  • Learning the similarity matrix: The student learns cross-modal correspondences through similarity matrices rather than being forced to reproduce the teachers’ embeddings exactly.The distillation loss compares the student matrix with the aggregated teacher matrix using a Huber loss.
  • Learning the similarity matrix: A simple element-wise mean aggregates teacher matrices, and directly learning the cross-modal similarity matrix is presented as a novel generalized distillation approach for this task.The method is compared with relationship-distillation alternatives in later experiments.
  • Teacher models: The teacher pool is created by training same-architecture retrieval models with different pretrained text embeddings, including mt grovle, openai-gpt, gpt2-large, gpt2-xl, and w2v.Up to five teachers are formed for each of four base student architectures.
  • Training and inference: TEACHTEXT adds only the distillation loss during training, with no additional trainable parameters or modalities in the final model.At test time, the teacher models are discarded, so the student retains the original inference configuration.

5. Experimental setup

The experiments evaluate TEACHTEXT across datasets, base architectures, teacher configurations, distillation variants, denoising, and video-side modality extensions. Results show broad gains, with no added inference overhead when multiple embeddings are distilled into one student.

  • Baseline improvements: Correcting compression artefacts, refining modalities, finetuning text embeddings, and changing the optimizer each improve the CE baseline, producing CE+.Text-embedding finetuning has particularly strong influence, while updated preprocessing improves fairness for retrained comparisons.
  • Text-embedding inference: TEACHTEXT outperforms direct concatenation and mean aggregation of multiple text embeddings without adding parameters or inference-time computational overhead.Using multiple embeddings directly nearly doubles CE+ parameters, whereas TEACHTEXT adds none.
  • Teacher variation: Adding teachers improves performance until a plateau after three teachers, leading the final experiments to use w2v, gpt2-xl, and gpt2-xl-F.The three selected teachers provide complementary text embeddings, including a target-dataset-finetuned gpt2-xl representation.
  • Distillation ablation: TEACHTEXT is more effective than relational and pairwise-distance distillation, while restricting distillation to top-K teacher predictions slightly reduces performance.The alternatives preserve intra-modal relationships, cross-modal distances, or only selected ranks rather than the full similarity matrix.
  • Method generality: TEACHTEXT improves performance across all tested base models and datasets, including over 5% absolute gains on DiDeMo and ActivityNet for MoEE, CE, and CE+.The comparison uses the geometric mean of R@1, R@5, and R@10 across multiple architectures and datasets.
  • Video-side modality extension: TEACHVIDEO extends distillation to multiple video modalities, improving over the student while retaining fewer modalities during inference.This extension is intended for settings with limited inference-time computational resources.

6. Conclusion

TEACHTEXT transfers complementary information from multiple pretrained text embeddings through teacher–student distillation for text-video retrieval. The paper reports state-of-the-art results on six benchmarks, a denoising application, and no additional inference-time computational cost.

  • 6. Conclusion: TEACHTEXT achieves state-of-the-art results on six video-retrieval benchmarks and supports denoising retrieval datasets.The method uses one or multiple teachers with different pretrained text embeddings.
  • 6. Conclusion: The method distills text-video similarity information from one or multiple teacher models into a student retrieval model.Teachers share the architecture while using different pretrained text embeddings.
  • 6. Conclusion: The supplementary material documents pretrained video and text embeddings, optimization settings, dataset protocols, and additional ablations.Video experts include action, object, face, audio, scene, speech, and OCR modalities; text embeddings include w2v, GPT, RoBERTa, ALBERT, and GPT2 variants.
  • 6. Conclusion: TEACHTEXT leverages multiple text encoders without adding inference-time computational cost.The approach leaves the number of model parameters unchanged, unlike concatenating text embeddings.

F. Dataset details

The experiments cover seven text-video retrieval datasets with distinct sizes, modalities, languages, and retrieval protocols. Ablations examine training configuration, similarity aggregation, denoising, distillation variants, and loss choices.

  • F. Dataset details: Seven video datasets are used to evaluate text-video retrieval, including MSRVTT, MSVD, DiDeMo, LSMDC, ActivityNet, VaTeX, and QuerYD.The datasets vary in scale, caption structure, languages, and retrieval protocols.
  • F. Dataset details: A batch size of 64 gives the best MSR-VTT performance in the reported batch-size ablation.This matches the batch size used by the corresponding method without TEACHTEXT.
  • F. Dataset details: Mean similarity-matrix aggregation is more effective than minimum or maximum aggregation and is used in the final TEACHTEXT algorithm.The comparison evaluates both standalone teacher performance and teacher integration.
  • F. Dataset details: Denoising is effective for retrieval datasets, with rank 40 selected for MSRVTT and a threshold of 100 evaluated on MSVD.The method filters training captions using teacher-assigned ranks, and denoising is excluded from other ablations.
  • F. Dataset details: Huber loss outperforms L1 loss and performs slightly better than L2 loss for distillation.Additional comparisons include direct embedding regression and angle-based relational distillation.

G.6. Mixture of architectures

The mixture-of-architectures ablation tests whether teachers with different underlying architectures improve TEACHTEXT. The preliminary results show no clear benefit from combining such architectures.

  • G.6. Mixture of architectures: TEACHTEXT normally assumes that teacher and student differ only in their pretrained text embeddings, whereas this ablation varies the underlying teacher architecture.All other ablations share the architecture between student and teacher.
  • G.6. Mixture of architectures: Mixtures of architectures provide no clear improvement when used as TEACHTEXT teachers.The experiment compares teachers with different underlying architectures, including groups of three architecture variations.
  • G.6. Mixture of architectures: The limited gain is attributed to architecture variants sharing similar video modalities, while greater modality diversity is expected to provide more additional information.The latter expectation is stated as a prospective interpretation rather than a demonstrated result.

G.7. Architecture extension

The architecture extension shows that TEACHTEXT remains effective across base models, including a lightweight CE-L variant. Its gains do not require adding text embeddings at test time or substantially increasing model parameters.

  • G.7. Architecture extension: Increasing training data strengthens the performance improvement brought by TEACHTEXT, and both w2v and GPT2-XL teachers improve over the nondistilled student.The teacher-type comparison reports a boost independent of the teacher’s nature.
  • G.7. Architecture extension: TEACHTEXT preserves most of the student’s correct retrievals while adding teacher-derived correct predictions on MSR-VTT.The figure compares three-teacher and single-teacher settings using Rank 1 retrieval.
  • G.7. Architecture extension: TEACHTEXT consistently improves retrieval across different base architectures, including the lightweight CE-L model.Table 10 reports effectiveness independent of the base architecture, while CE-L substantially reduces parameter count.
  • G.7. Architecture extension: Qualitative examples show TEACHTEXT correcting the student’s retrieval prediction using information from the teacher.The displayed cases report teacher and student ranks and similarities for the ground-truth video.
  • G.7. Architecture extension: The proposed approach leaves the parameter count unchanged, whereas concatenating text embeddings adds approximately 240M learnable parameters.The reported CE+ model sizes are 503.98M with concatenation versus 262.73M for the comparison.

G.9. Amount of training data vs performance.

TEACHTEXT’s advantage increases as more training data is used, while even a weak w2v teacher provides useful complementary information.

  • The performance gap from TEACHTEXT increases as the amount of training data grows.This suggests the method may remain useful on larger-scale datasets.
  • Although w2v alone underperforms the student without TEACHTEXT, distillation from a w2v teacher produces a significant gain.The result indicates that the teacher embedding contains exploitable information despite its lower standalone performance.
  • A different text embedding contributes additional information that TEACHTEXT can exploit.

G.11. Influence of distillation over the correctly retrieved samples

On MSR-VTT, TEACHTEXT produces substantial overlap between correctly retrieved samples from the student and teacher, with results shown for both multiple and single teachers.

  • TEACHTEXT yields a significant share of correctly retrieved samples shared with the teacher on MSR-VTT.The comparison uses R1 shares for students trained with and without TEACHTEXT and for the teacher.

H. Comparison to prior work

Across extensive comparisons with prior methods, TEACHTEXT improves retrieval while preserving the base architecture’s parameter count and performing well in a lightweight regime.

  • TEACHTEXT brings a clear improvement across comparisons with methods from the literature while keeping the total parameter count equal to the base architecture.The comparisons cover multiple benchmarks and include v2t results where reported.
  • The comparison tables cover MSR-VTT, MSVD, DiDeMo, LSMDC, ActivityNet, VaTeX, and QuerYD benchmarks.
  • A lightweight architecture using w2v has very good results when combined with TEACHTEXT.This demonstrates effectiveness across different parameter regimes.
Loading 2104.08271v2…