Source-linked AI summary
Dual Encoding for Zero-Example Video Retrieval
Jianfeng Dong, Xirong Li, Chaoxi Xu, Shouling Ji, Yuan He, Gang Yang, Xun Wang
TL;DR
Zero-example video retrieval must match natural-language queries with videos without visual examples, while concept-based representations face concept-selection and classifier challenges. The paper proposes concept-free dual multi-level encoding for videos and queries before common-space learning, and reports state-of-the-art performance across three benchmarks.
Problem
Zero-example video retrieval seeks to match natural-language queries with videos when no visual example is provided, while concept-based methods face difficulties specifying, detecting, and selecting suitable concepts.
Method
The paper uses dual multi-level encoding with mean pooling, recurrent networks, convolutional networks, and skip connections to form dense representations before common-space learning.
Results
The proposed dual encoding establishes state-of-the-art performance on MSR-VTT and TRECVID 2016 and 2017 Ad-hoc Video Search benchmarks.
Takeaways & Limitations
Dual encoding supports concept-free sequence-to-sequence cross-modal matching by progressively representing global, local, and temporal patterns in videos and sentences.
Takeaways & Limitations
TRECVID ground truth is partially available at shot level and may underestimate scores for results outside the manually verified pool.
Abstract
from arXiv · showhide
This paper attacks the challenging problem of zero-example video retrieval. In such a retrieval paradigm, an end user searches for unlabeled videos by ad-hoc queries described in natural language text with no visual example provided. Given videos as sequences of frames and queries as sequences of words, an effective sequence-to-sequence cross-modal matching is required. The majority of existing methods are concept based, extracting relevant concepts from queries and videos and accordingly establishing associations between the two modalities. In contrast, this paper takes a concept-free approach, proposing a dual deep encoding network that encodes videos and queries into powerful dense representations of their own. Dual encoding is conceptually simple, practically effective and end-to-end. As experiments on three benchmarks, i.e. MSR-VTT, TRECVID 2016 and 2017 Ad-hoc Video Search show, the proposed solution establishes a new state-of-the-art for zero-example video retrieval.
1. Introduction
Zero-example video retrieval requires matching natural-language queries with videos without visual examples, while existing concept-based methods face representation and concept-selection difficulties. The paper proposes concept-free dual multi-level encoding to learn dense representations for both modalities before common-space matching.
- Zero-example retrieval matches natural-language queries to videos without providing a visual example.
- Concept-based methods depend on selecting relevant, detectable concepts and training classifiers for video and query representations.
- The paper learns common semantic embeddings directly from videos and queries through a concept-free approach.
- Dual encoding jointly exploits mean pooling, recurrent networks, convolutional networks, and skip connections to build progressively higher-level representations.
- The model combines multi-level video and text encodings before common-space learning, rather than treating cross-modal matching as the entire task.
- Combined with common-space learning, dual encoding establishes a new state-of-the-art on three challenging zero-example video-retrieval benchmarks.
2. Related Work
Prior zero-example video-retrieval systems largely use concept classifiers, linguistic rules, or a single encoding strategy. This paper instead progressively exploits global, local, and temporal patterns in both videos and sentences through dual multi-level encoding.
- The TRECVID Ad-hoc Video Search challenge evaluates zero-example video retrieval without training examples for the target events.
- Many leading systems use visual concept classifiers for videos and linguistic rules to extract concepts from textual queries.
- Common-space retrieval methods typically aggregate video-frame features by mean pooling, while alternatives use recurrent or convolutional networks.
- Existing query encoders include bag-of-words, recursive networks, LSTMs, bidirectional LSTMs, and GRUs.
- The paper claims to be the first work explicitly and progressively exploiting global, local, and temporal patterns in both videos and sentences.
- The proposed network performs parallel multi-level encodings and projects combined video and sentence vectors into a common space for similarity computation.
3. The Dual Encoding Network
The dual encoding network builds multi-level representations for videos and text before common-space learning. It combines global, temporal, and local information progressively, then applies the same overall encoding design to both modalities.
- Overview: Videos and sentences are encoded separately into multi-level representations before their outputs are learned in a shared space.The encoding results combine modality-specific sequence items in a coarse-to-fine fashion.
- Level 1: Global Encoding by Mean Pooling: Mean pooling provides a global video representation by averaging frame-level features and capturing repeatedly present visual patterns.The method uniformly samples video frames and extracts pretrained CNN features before pooling.
- Level 2: Temporal-Aware Encoding by biGRU: A biGRU processes frame features in forward and reverse order to model temporal information using both past and future context.Its forward and backward hidden states are concatenated and then mean-pooled into an encoding.
- Level 3: Local-Enhanced Encoding by biGRU-CNN: A multi-scale 1-D CNN over biGRU outputs enhances local patterns by applying filters of sizes 2, 3, 4, and 5 before max pooling and concatenation.Different filter sizes allow adjacent rows to interact over multiple local scales.
- Multi-level Video Encoding: The three video encoding levels are concatenated because their outputs are treated as complementary despite some redundancy.This produces a multi-level video encoding from the global, temporal, and local representations.
- Text-side Multi-level Encoding: The video-side network is adapted for text using bag-of-words, biGRU, and biGRU-CNN encodings, whose outputs are concatenated into a sentence representation.Words are embedded before recurrent processing, while the text CNN uses filter sizes 2, 3, and 4.
4. Common Space Learning
Common-space learning projects the independently encoded video and sentence representations into comparable vectors. The network is trained end-to-end with cosine similarity and a hardest-negative marginal ranking loss.
- Common Space Learning: VSE++ is selected as the common-space learning algorithm because it was effective for image-text retrieval and video retrieval, with publicly available source code.The paper builds its common-space learner on this existing algorithm.
- Projection: Affine transformations followed by batch normalization project encoded videos and sentences into a common space.The modality-specific fully connected layers use W_v, W_s and bias terms b_v, b_s.
- Similarity: Cosine similarity between the projected video and sentence vectors defines the video-text matching score.The similarity is computed with respect to the trainable parameters θ.
- Training Objective: The improved marginal ranking loss penalizes relevant pairs against the most similar negative sentence and video in the current mini-batch.The loss uses a margin α and jointly trains the dual encoding and common-space networks.
5. Evaluation
Across video-retrieval benchmarks and related image-text experiments, the proposed dual encoding method is evaluated against state-of-the-art methods, encoding variants, training data, and retrieval settings. It performs best in the reported comparisons, while enabling independent offline video processing and online query retrieval.
- MSR-VTT: The proposed method performs best on MSR-VTT, while adapting W2VV to the improved marginal ranking loss raises its Sum of Recalls from 90.3 to 132.1.The comparison includes text-to-video and video-to-text retrieval; the table caption identifies larger recalls and mAP and smaller median rank as better.
- Ablation study: The full Level 1 + 2 + 3 multi-level encoding is best, and adding encoding layers improves overall performance because the combined layers are complementary.Among individual levels, biGRU-CNN is most effective; models combining outputs from previous layers outperform the first three single-level models.
- Ablation study: Single-side encoding obtains Sum of Recalls of 143.6 for video-side encoding and 137.1 for text-side encoding, supporting the necessity of dual encoding.The reported scores also suggest that video-side encoding is more beneficial than text-side encoding.
- TRECVID AVS: On TRECVID 2016 and 2017, the proposed method performs best with infAP of 0.159 and 0.208, exceeding the cited single-model and W2VVimrl comparisons.A competing 2017 ensemble reaches 0.206 using ten models, while its best single model reaches 0.150 with the same ResNeXt-101 feature.
- Generalization and efficiency: The model performs best in the reported cross-dataset and cross-domain evaluations, improves VSE++ on Flickr30K and MSCOCO, and retrieves from 335,944 IACC.3 videos in approximately 0.14 second.The network encodes videos and sentences independently after training, allowing offline video processing and on-the-fly ad-hoc query answering.
6. Summary and Conclusions
The paper proposes dual encoding for zero-example video retrieval, using multi-level encodings to represent videos and natural-language queries as dense representations before common-space matching. Experiments support dual encoding and identify biGRU-CNN and video-side multi-level encoding as particularly effective.
- Summary and Conclusions: Dual encoding represents videos and natural-language queries as powerful dense representations before common-space learning for cross-modal matching.The network jointly exploits multiple encoding strategies at different levels.
- Summary and Conclusions: Among the three encoding levels, biGRU-CNN is the most effective when used alone.It builds a one-dimensional convolutional network on top of a bidirectional GRU.
- Summary and Conclusions: Video-side multi-level encoding is more beneficial than its text-side counterpart, and the authors recommend dual encoding for state-of-the-art performance.The paper also identifies potential application to video question answering requiring video/text encoding.