Source-linked AI summary
Word2VisualVec: Image and Video to Sentence Matching by Visual Feature Prediction
Jianfeng Dong, Xirong Li, Cees G. M. Snoek
TL;DR
The paper addresses image- and video-to-sentence matching, where prior approaches use a joint latent subspace. Word2VisualVec predicts deep visual features from vectorized sentences and matches in visual space; it reports state-of-the-art results across four image and video benchmarks.
Problem
Image- and video-to-sentence matching requires comparing distinct modalities, and prior approaches rely on a joint latent subspace.
Source: §1 · p. 1Method
Word2VisualVec uses sentence vectorization and a multilayer perceptron to predict deep visual features, matching sentences to images or videos in visual space.
Source: §1 · pp. 1–2, §2.2 · p. 3Results
The paper reports state-of-the-art results for image matching on Flickr8k and Flickr30k and video matching on MSVD and the TrecVid 2016 Video-to-Text benchmark.
Source: §5 · p. 8Takeaways & Limitations
The reported results cover both image and video matching, and the paper demonstrates the visual space's potential for text retrieval.
Source: §5 · pp. 8–9Greater network depth increases learning capacity but can produce solutions that minimize training loss while generalizing suboptimally to unseen test data.
Source: §3.1 · p. 4
Abstract
from arXiv · showhide
This paper strives to find the sentence best describing the content of an image or video. Different from existing works, which rely on a joint subspace for image / video to sentence matching, we propose to do so in a visual space only. We contribute Word2VisualVec, a deep neural network architecture that learns to predict a deep visual encoding of textual input based on sentence vectorization and a multi-layer perceptron. We thoroughly analyze its architectural design, by varying the sentence vectorization strategy, network depth and the deep feature to predict for image to sentence matching. We also generalize Word2VisualVec for matching a video to a sentence, by extending the predictive abilities to 3-D ConvNet features as well as a visual-audio representation. Experiments on four challenging image and video benchmarks detail Word2VisualVec's properties, capabilities for image and video to sentence matching, and on all datasets its state-of-the-art results.
1. Introduction
The paper seeks the sentence that best describes an image or video, matching in visual space rather than a joint latent subspace. Word2VisualVec predicts deep visual features from sentence representations and extends this approach to video.
1. Introduction
- The approach matches images and videos to sentences in visual space only, unlike prior methods that use a latent subspace.The paper describes this as its first image-to-sentence matching approach in visual space only.Source: §1 · pp. 1–2
- Word2VisualVec transforms vectorized text through a multilayer perceptron to predict deep visual features for image or video sentence matching.It can use word2vec embeddings and is also compatible with bag-of-words and hashing.Source: §1 · pp. 1–2
- Experiments report state-of-the-art results on Flickr8k, Flickr30k, MSVD, and the TrecVid 2016 Video-to-Text benchmark.The paper also investigates the visual space's potential for text retrieval.Source: §1 · p. 2
- For video matching, the approach predicts 3-D convolutional features and a visual-audio representation that includes Mel Frequency Cepstral Coefficients.The paper considers several visual features and describes the prediction approach as general to any deep visual feature it is trained on.Source: §1 · p. 2
2. Related Work
Prior image- and video-to-sentence approaches typically map text and visual inputs into a shared representation. Word2VisualVec instead projects vectorized sentences into visual feature space, while allowing different sentence-vectorization strategies.
2.1. Sentence vectorization
- Word hashing represents sentences with letter-trigram counts and scales better than bag-of-words because it uses fewer unique trigrams than English words.It can generalize to infrequent and unseen words.Source: §2.1 · p. 2
- Word2VisualVec accepts bag-of-words, hashing, or word2vec as its sentence-vectorization input, unlike methods tied to a particular representation.Fixed-length sentence vectors are a prerequisite for matching variable-length sentences; word2vec embeddings can be mean-pooled, while word hashing counts letter trigrams.Source: §2.1 · p. 2
2.2. Matching images and videos to sentences
- Earlier image-to-sentence work used Kernel CCA or projected deep image features into shorter vectors, while language models transformed sentence representations with LSTMs or RNNs.Directly feeding visual features into an LSTM or RNN would increase trainable parameters in quadratic order.Source: §2.2 · p. 2
- Prior video-to-sentence approaches used shared-subspace embeddings, recurrent models, or attention to connect video features with sentence representations.The cited approaches include aggregating subject-verb-object vectors and pooling frame features.Source: §2.2 · pp. 2–3
- Word2VisualVec leaves visual features unprojected and projects the vectorized sentence into visual space, based on the hypothesis that deep visual features already support image or video comparison.Its network maps sentence vectors through an MLP to a chosen visual feature representation.Source: §2.2 · p. 3, Figure 2 · p. 3
3. Word2VisualVec
Word2VisualVec maps sentences into the visual feature space used to represent images or videos, so their relevance can be computed there. It vectorizes text, predicts a visual representation with an MLP, and uses those predicted features for matching.
3. Word2VisualVec
- Word2VisualVec produces a sentence representation in the same visual feature space as an image, where their similarity can be computed.The sentence representation is designed to be compared with a ConvNet feature vector.Source: §3 · p. 3
3.1. Architecture
- The architecture supports bag-of-words, word hashing, or word2vec inputs, then transforms the sentence vector through MLP layers into a visual feature.Word hashing uses letter trigrams; the word2vec strategy averages word embeddings, trained here on Flickr image tags.Source: §3.1 · p. 3
- Adding layers increases model capacity but can yield solutions that minimize training loss while performing suboptimally on unseen data.The paper studies how deep the network can be without losing generalization ability.Source: §3.1 · p. 4
3.2. Learning algorithm
- The learning objective reconstructs an image's ConvNet feature from its paired sentence using MSE, which performed better than the tested marginal ranking loss.The paper links visual-space sentence representations with the ability to capture correlations between differently worded descriptions of similar content.Source: §3.2 · p. 4
3.3. Image to sentence
- For image-to-sentence matching, the model predicts each candidate sentence's visual encoding and ranks sentences by cosine similarity to the image feature.The authors report cosine similarity performs better than the dot product.Source: §3.3 · p. 4
3.4. Video to sentence
- For videos, Word2VideoVec predicts a video feature from text, using mean-pooled frame features and optionally concatenated visual and audio features.The visual features can come from image ConvNets or 3-D ConvNets; the audio representation is a bag of quantized MFCCs.Source: §3.4 · pp. 4–5
4. Experiments
Across image and video benchmarks, Word2VisualVec and Word2VideoVec compare favorably with recent alternatives. Experiments also identify architecture and input settings, and show that the learned visual space can support text-only retrieval.
4.1. Properties of Word2VisualVec
- Predicting deeper ConvNet features improves matching performance, and ResNet-152 yields the best performance among the tested visual features.The comparison uses a three-layer Word2VisualVec across the features.Source: §4.1 · p. 5, Table 1 · p. 5
- The three-layer 500-1000-2048 model peaks on Flickr8k; performance declines beyond three layers as overfitting risk increases.The authors describe this depth as balancing model capacity and generalization.Source: §4.1 · p. 5, Table 2 · p. 5
- Bag-of-words performs best on each dataset and Flickr30k-to-Flickr8k transfer, while hashing and word2vec do better for Flickr8k-to-Flickr30k transfer.The latter result indicates better generalization when training data is limited.Source: §4.1 · p. 6, Table 3 · p. 6
- Performance mostly converges with half the Flickr8k training data, while word2vec beats alternatives at 1k and 2k images; the recommended input is bag-of-words with adequate data and word2vec when data is scarce.The 1k, 2k, and 4k subsets were created by random downsampling, with each size tested five times.Source: §4.1 · p. 6
4.2. Potential of Visual Space for Text Retrieval
- Word2VisualVec achieves higher mAP than bag-of-words and word2vec on both text-retrieval test sets, with comparatively limited benefit from stop-word removal.The model was used directly from the preceding experiment without retraining.Source: §4.2 · p. 6, Table 4 · p. 7
4.3. Video-to-Sentence
- Word2VideoVec outperforms competing models on MSVD; ResNet-152 is its strongest visual feature, and word2vec beats bag-of-words as input.The MSVD videos were muted, so audio could not be evaluated.Source: §4.3 · p. 7, Table 5 · p. 7
4.4.1 Image to Sentence
- Word2VisualVec outperforms recent image-to-sentence alternatives on Flickr8k and Flickr30k, even when a competitor uses additional region annotations.The authors argue this supports direct matching in deep visual feature space.Source: §4.4.1 · p. 7, Table 6 · p. 7
4.4.2 Video to Sentence
- Mean Inverted Rank ranges from 0.097 to 0.110, leading both TrecVid 2016 test sets among submissions from seven teams; visual-audio prediction improves results further.The benchmark uses blind testing, and the model was trained on MSR-VTT pairs because the provided TrecVid training set was considered insufficient.Source: §4.4.2 · pp. 7–8, Figure 3 · p. 8
5. Conclusions
Word2VisualVec uses deep visual features as a shared representation for image- and video-to-sentence matching, and the paper reports state-of-the-art results across four benchmarks. It also demonstrates visual-space text retrieval potential.
5. Conclusions
- Word2VisualVec uses deep visual features as a shared representation for image-to-sentence matching.The conclusion also advocates this representation for video-to-sentence matching.Source: §5 · p. 8
- The paper reports state-of-the-art results on Flickr8k and Flickr30k for image matching and on MSVD and TrecVid 2016 for video matching.Source: §5 · p. 8
- The authors demonstrate the potential of visual space for text retrieval.Source: §5 · p. 9