PaperlayerView exact revision on arXiv

Source-linked AI summary

Improving Video-Text Retrieval by Multi-Stream Corpus Alignment and Dual Softmax Loss

Xing Cheng, Hezheng Lin, Xiangyu Wu, Fan Yang, Dong Shen

arXiv:2109.04290v3cs.CV

TL;DR

Video-text retrieval models face structural and content heterogeneity that can cause overfitting and one-way matching. The paper proposes CAMoE for multi-stream corpus alignment and Dual Softmax loss for dual optimal matching, achieving stronger benchmark performance with little extra training burden.

  • Problem

    Structural and content heterogeneity between video and text makes CLIP-based video-text retrieval models prone to overfitting and can produce confusing one-way matching.

  • Method

    CAMoE uses separate fusion, entity, and action experts with a gate to integrate their representations, while Dual Softmax loss addresses content heterogeneity through dual optimal matching.

  • Results

    Dual Softmax loss improves all metrics across benchmarks and substantially raises performance when applied to CAMoE and other models.

  • Takeaways & Limitations

    CAMoE reaches SOTA for structural heterogeneity, while Dual Softmax loss achieves significant improvement with little extra training burden and may extend to other cross-modal tasks.

  • Takeaways & Limitations

    Adding the gate to entity and action tasks can overfit their matching, so the gate is used only by the fusion expert.

Abstract

from arXiv · show

Employing large-scale pre-trained model CLIP to conduct video-text retrieval task (VTR) has become a new trend, which exceeds previous VTR methods. Though, due to the heterogeneity of structures and contents between video and text, previous CLIP-based models are prone to overfitting in the training phase, resulting in relatively poor retrieval performance. In this paper, we propose a multi-stream Corpus Alignment network with single gate Mixture-of-Experts (CAMoE) and a novel Dual Softmax Loss (DSL) to solve the two heterogeneity. The CAMoE employs Mixture-of-Experts (MoE) to extract multi-perspective video representations, including action, entity, scene, etc., then align them with the corresponding part of the text. In this stage, we conduct massive explorations towards the feature extraction module and feature alignment module. DSL is proposed to avoid the one-way optimum-match which occurs in previous contrastive methods. Introducing the intrinsic prior of each pair in a batch, DSL serves as a reviser to correct the similarity matrix and achieves the dual optimal match. DSL is easy to implement with only one-line code but improves significantly. The results show that the proposed CAMoE and DSL are of strong efficiency, and each of them is capable of achieving State-of-The-Art (SOTA) individually on various benchmarks such as MSR-VTT, MSVD, and LSMDC. Further, with both of them, the performance is advanced to a big extend, surpassing the previous SOTA methods for around 4.6\% R@1 in MSR-VTT.

Introduction

The paper addresses structural and content heterogeneity in video-text retrieval with multi-stream alignment and Dual Softmax correction. It decomposes visual-semantic information into specialized tasks and revises similarity scores for dual optimal matching.

  • Introduction: CAMoE decomposes video-text retrieval into specialized semantic tasks and aligns distinct video representations with corresponding textual information.The design uses experts for aspects such as entity and action, with a gate integrating representations.
  • Introduction: Content heterogeneity arises because explicit videos may correspond to unspecific or fuzzy text, while standard softmax ignores cross-retrieval information.This can make one text appear maximally similar to multiple videos and produce confusing similarity results.
  • Introduction: DSL revises the similarity matrix by multiplying the original scaled similarity matrix with a cross-direction prior probability matrix.The dual optimal-match hypothesis states that an optimal match in one retrieval direction should correspond to the highest score in the symmetric direction.
  • Introduction: The paper proposes a visual-semantic data-decomposing and multi-task construction scheme that may extend to image-text generation, image-text retrieval, and image captioning.The contribution is presented as applicable beyond video-text retrieval.
  • Introduction: The proposed CAMoE and Dual Softmax loss advance the state of the art, with DSL described as an efficient performance-improvement method.The paper also identifies contents heterogeneity as an area for further work.

Related work

Prior video-text retrieval methods either impose designed alignment structures or directly match video and text features in embedding space. Recent large-model approaches favor two-stream architectures and CLIP-based visual representations.

  • Related work: Video-text retrieval methods are broadly categorized as alignment-based or embedding-based according to how they improve performance.This distinction organizes prior approaches by whether they structure modalities before similarity calculation or match features directly.
  • Related work: Alignment-based methods decompose video and text into regular or semantic structures to facilitate similarity calculation.Examples include visual words, semantic concepts, and sentence partitions into events, actions, and entities.
  • Related work: Embedding-based methods emerged with large datasets and models, reducing dependence on designer-defined regulation schemes.These methods directly match video and text features in embedding space.
  • Related work: CLIP-based visual encoders became prominent after comparisons found CLIP performed best among adopted pre-training models.CLIP4Clip additionally models temporal dependency between video frames using different similarity calculators.

Method

The method converts video and text into aligned multi-stream representations using task-specific experts, frame aggregation, and sentence-generation strategies. It further applies Dual Softmax Loss to revise similarity scores using cross-direction priors and filter one-sided matches.

  • Overall Architecture: CAMoE converts video and text into three streams for consistency learning across sentence, entity, and action matching.Text nouns and verbs are transformed through sentence generation strategies, while visual experts learn distinct semantic matching tasks.
  • Experts and Gating network: CLIP-pretrained ViT provides bottom visual features, while fusion, entity, and action experts learn distinct semantic matches.A gate integrates entity and action representations with fusion representations, improving the fusion expert.
  • Sentence generation strategy: Three sentence generation strategies—RKW, AKWE, and MUW—construct semantic representations from keywords or masked text.MUW masks non-keywords and uses the CLS token, whereas RKW recombines keywords and AKWE averages keyword output embeddings.
  • Visual Frames Aggregation Scheme: Three visual frame aggregation schemes—mean pooling, squeeze-and-excitation attention, and self-attention—serve different experts or gates.The best configuration uses squeeze-and-excitation attention for the gating network and fusion expert, with self-attention for entity and action experts.
  • Experts and Gating network: Only the fusion expert uses a gating module because gating entity and action tasks causes overfitting.The authors attribute this to the whole-sentence information available to the fusion expert making the other tasks too simple.
  • Loss function: Dual Softmax Loss multiplies a cross-direction prior matrix with the original similarity matrix to filter one-sided matches.It highlights pairs with strong Text-to-Video and Video-to-Text probabilities, producing a more convincing match.

Experiments

Experiments evaluate CAMoE and Dual Softmax across MSR-VTT, MSVD, and LSMDC using standard retrieval metrics. CAMoE and Dual Softmax improve retrieval performance, with Dual Softmax further strengthening results across benchmarks.

  • Datasets and Metrics: Experiments use MSR-VTT, MSVD, and LSMDC with Recall at rank K, median rank, and mean rank as evaluation metrics.Higher R@K and lower MdR and MnR indicate better performance.
  • Implementation: The implementation samples 16 frames per video and uses 512-dimensional visual and semantic embeddings with CLIP-based BERT and ViT components.BERT and ViT use a learning rate of 1e-7, while other parameters use 1e-4.
  • MSR-VTT: CAMoE surpasses previous SOTA on MSR-VTT with the standard loss, while Dual Softmax produces further gains.The reported R@1 increments are approximately 2.8% and 6.4%, respectively.
  • MSVD: 46.9 Text-to-Video R@1 is achieved by CAMoE on MSVD, establishing a new SOTA, while prediction mean rank decreases by 0.2%.
  • LSMDC: 4.3% improvement is obtained on LSMDC with Dual Softmax, compared with 0.9% for CAMoE alone.LSMDC contains many videos, each paired with only one caption, and neural networks generally perform poorly on this dataset.

Ablation Studies

Ablation studies identify the most effective architecture, sentence generation strategy, frame aggregation scheme, and Dual Softmax behavior. Distinct multi-task inputs with a single gate and specialized aggregation choices perform best, while Dual Softmax improves results broadly.

  • Structure Design: Multi-task inputs with distinct captions and a single gate outperform single-task and same-caption multi-task alternatives on MSR-VTT.Adding gates to all tasks is not recommended when the tasks have subordinate relationships.
  • Sentence Generation: RKW disrupts sentence organization, whereas AKWE uses the whole sentence and causes overfitting rather than teaching experts specific features.
  • Frame Aggregation: CAMoE performs best when gate and fusion experts use squeeze-and-excitation attention, while entity and action experts use self-attention.Self-attention supports complex embedding transformations, whereas squeeze-and-excitation attention adds few parameters and learns keyframe emphasis.
  • Loss Optimization: Dual Softmax improves all reported metrics across all benchmarks and provides gains when applied to CLIP, FROZEN, and CLIP4Clip.The gain is around 10 points for CLIP V2T-R@1 and 4.9% for CLIP4Clip.

Quantitative Analysis and Visualization

Expert-level analysis shows that the fusion expert contributes most, while entity and action experts provide additional information. Visualization attributes Dual Softmax’s effect to filtering outliers and sharpening confident similarity points.

  • Expert Importance Analysis: The fusion expert performs best, entity expert reaches about 27%, and action expert reaches 8.4% for T2V and 4.2% for V2T.
  • Expert Importance Analysis: The gate assigns average weights of 0.63, 0.29, and 0.08 to the three experts on the MSR-VTT test set.The reported weights indicate that the two additional experts affect the combined representation.
  • Dual Softmax Visualization: Dual Softmax improves the inferred probability matrix by filtering outliers and sharpening crucial, high-confidence points.The comparison is between the original inference method and Dual Softmax.
  • Dual Softmax Visualization: The cross-direction prior suppresses samples with high Video-to-Text similarity but low Text-to-Video probability.This correction filters borderline cases and sharpens more convincing matches.

Conclusion

The paper identifies structural and content heterogeneity in video-text retrieval and proposes CAMoE and Dual Softmax loss to address them. These methods advance state-of-the-art performance and support broad application potential.

  • CAMoE addresses structural heterogeneity in video-text retrieval and reaches state-of-the-art performance.
  • Dual Softmax loss addresses errors from confusing sentences that may match multiple videos, improving performance with little extra training burden.
  • The proposed methods are presented as having a wide range of potential application scenarios in industry and academia.

The Demonstration of Preventing Overfitting

The experiments compare loss evolution for single-task and multi-task training and testing. Their contrasting loss patterns indicate that the proposed multi-task design can prevent overfitting.

  • Multi-task training maintains higher training loss than test loss, unlike single-task training, whose training loss rapidly falls below test loss.
  • The contrasting loss evolution indicates that the proposed multi-task design can prevent model overfitting.

Different Videos

The gating module assigns different expert weights across videos. Entity and action experts become more influential under distinct visual conditions.

  • Expert weights vary substantially across videos, indicating that the gate assigns diverse videos different expert contributions.
  • Entity experts are relatively more critical when conspicuous objects or homogeneous entities dominate the video.
  • Action expert weights exceed their 0.08 average when videos contain apparent entities performing behaviors or have differing contents and perspectives.

Further Experiments

Additional experiments cover MSR-VTT full, DiDeMo, and Activitynet. The proposed method is reported to generalize strongly and achieve state-of-the-art performance across various datasets.

  • The further experiments evaluate MSR-VTT full, DiDeMo, and Activitynet.
  • MSR-VTT full uses 7k training videos and 3k test videos, while DiDeMo contains about 10,000 videos ranging from 12 to 429 seconds.
  • The results show strong generalization and state-of-the-art performance across various datasets.
Loading 2109.04290v3…