Source-linked AI summary
T2VLAD: Global-Local Sequence Alignment for Text-Video Retrieval
Xiaohan Wang, Linchao Zhu, Yi Yang
TL;DR
Text-video retrieval must match natural-language descriptions with relevant videos, yet many methods emphasize global similarity and overlook local details. T2VLAD uses shared semantic centers for efficient local alignment alongside complementary global alignment and supervision. The method achieves consistent improvements on three standard benchmarks and clearly outperforms the state of the art.
Problem
Text-video retrieval requires measuring similarities between descriptions and videos, but global-only methods overlook local details while complex local matching is computationally expensive.
Method
T2VLAD adaptively aggregates multimodal video sequences and text features with shared semantic centers for local alignment, while adding complementary global alignment.
Results
T2VLAD achieves consistent improvements on MSRVTT, ActivityNet Captions, and LSMDC, clearly outperforming the state of the art.
Takeaways & Limitations
Local semantic alignment is critical for high-performance text-video retrieval, and global alignment provides complementary information and additional supervision.
Abstract
from arXiv · showhide
Text-video retrieval is a challenging task that aims to search relevant video contents based on natural language descriptions. The key to this problem is to measure text-video similarities in a joint embedding space. However, most existing methods only consider the global cross-modal similarity and overlook the local details. Some works incorporate the local comparisons through cross-modal local matching and reasoning. These complex operations introduce tremendous computation. In this paper, we design an efficient global-local alignment method. The multi-modal video sequences and text features are adaptively aggregated with a set of shared semantic centers. The local cross-modal similarities are computed between the video feature and text feature within the same center. This design enables the meticulous local comparison and reduces the computational cost of the interaction between each text-video pair. Moreover, a global alignment method is proposed to provide a global cross-modal measurement that is complementary to the local perspective. The global aggregated visual features also provide additional supervision, which is indispensable to the optimization of the learnable semantic centers. We achieve consistent improvements on three standard text-video retrieval benchmarks and outperform the state-of-the-art by a clear margin.
1. Introduction
Text-video retrieval seeks relevant videos from natural-language descriptions, but global-only methods overlook fine-grained semantic alignment. T2VLAD addresses this with efficient shared semantic topics, local alignment, and complementary global supervision.
- Text-video retrieval embeds descriptions and videos into a joint space, while global-only comparisons overlook fine-grained semantic alignment.
- T2VLAD automatically learns shared semantic topics to summarize text and video features jointly during end-to-end training.Shared text and video topic representations reduce the semantic gap between modalities.
- T2VLAD locally aligns grouped text and video features within the same topics instead of using complex pairwise local matching.The method implements this alignment with a shared-center NetVLAD-based encoding.
- Global alignment computes similarity between aggregated video features and global text features, complementing local alignment and supervising semantic-topic optimization.
2. Related Work
Related work spans multimodal text-video retrieval, language representation learning, and VLAD-based feature aggregation. T2VLAD uses VLAD for local cross-modal similarity matching rather than downstream classification.
- Text-video retrieval is more challenging than text-image retrieval because videos contain temporal dynamics and richer multimodal information.
- Prior retrieval methods use joint visual-semantic embeddings, hard-negative mining, and multimodal video features such as RGB, motion, and audio.
- BERT and transformer-based sequence encoders are used to strengthen contextual language representations in vision-and-language retrieval systems.
- Unlike typical NetVLAD applications for discriminative feature learning or classification, T2VLAD applies VLAD to local text-video similarity matching.It introduces a local alignment loss rather than classifying the aggregated features.
3. Method
T2VLAD aligns text and video globally and locally by combining multi-expert video features, BERT text features, shared semantic centers, and complementary global similarities.
- Video Representations: T2VLAD extracts segment-level video representations from multiple pretrained experts to capture modality-specific information for global-local alignment.Each expert produces representations for temporal video segments.
- Video Representations: A one-layer self-attention module fuses projected multi-expert video features while preserving their locality and reducing computation.Unlike deeper transformer-based aggregation, the output retains the input feature length.
- Text Representations: BERT produces contextual word embeddings, while text global features are extracted jointly with local representations for T2VLAD.The BERT model is optimized end-to-end with the other framework modules.
- Local Alignment: Shared learnable centers cluster local text and video features into aligned topic representations, enabling cosine-based local similarity measurement.A K+1th center represents background information, whose aggregated feature is excluded from similarity computation.
- Global Alignment: Global alignment aggregates expert-specific video features and compares them with corresponding projected text features using a weighted sum of cosine distances.The expert weights are generated from the text representation through linear projection and softmax normalization.
4. Experiments
Experiments evaluate T2VLAD across three text-video retrieval benchmarks, efficiency measurements, ablations, and qualitative visualizations. Results show consistent gains over strong baselines, while ablations support the complementary roles of global-local alignment and shared VLAD centers.
- Benchmark comparisons: T2VLAD consistently improves state-of-the-art performance on MSRVTT, ActivityNet Captions, and LSMDC retrieval benchmarks.The evaluations cover both text-to-video and video-to-text retrieval where reported.
- Benchmark comparisons: 5.8% R@1 gain over MMT on MSRVTT 1k-B text-to-video retrieval raises performance from 20.3% to 26.1%.For video-to-text retrieval on the same split, T2VLAD improves R@1 from 21.1% to 26.7%, a 5.6% gain.
- Benchmark comparisons: 2.9% R@1 improvement over MMT + HT pretrain is achieved on MSRVTT 1k-A text-to-video retrieval without additional training videos.The paper reports a clear-margin advantage across all metrics on this split.
- Efficiency: 0.4s is required to encode 1k MSRVTT videos on a single V100 GPU, compared with 1.1s for MMT.The reported T2VLAD time excludes expert encoding.
- Ablation study: Removing local alignment substantially lowers performance, while removing global alignment at test time reduces text-to-video R@1 by 5.2%.Training only the local alignment branch causes the loss not to converge, and the global feature is reported as complementary to local information.
- Ablation study: Shared VLAD centers outperform separate centers, and replacing video VLAD with max pooling produces lower performance than T2VLAD.The findings support joint VLAD encoding and center sharing as beneficial for reducing the text-video semantic gap.
- Qualitative results: Assignment visualizations show text and video features grouped around shared centers, while retrieval examples show improved alignment for multiple semantic topics such as “parks”.T2VLAD retrieves the ground-truth video in the examples, whereas the model without local alignment returns less precise results.
5. Conclusion
The paper introduces an end-to-end text-video sequence alignment method based on T2VLAD and NetVLAD. Results on three standard benchmarks demonstrate its effectiveness, while visualization supports joint semantic topic learning.
- The paper introduces an end-to-end text-video sequence alignment method using NetVLAD-based T2VLAD for collaborative text-video encoding.The method targets local semantic alignment between texts and videos.
- Results on three standard text-video retrieval benchmarks clearly demonstrate the effectiveness of the proposed method.
- Visualization results validate the motivation for jointly learning semantic topics across text and video features.