PaperlayerView exact revision on arXiv

Source-linked AI summary

Predicting Visual Features from Text for Image and Video Caption Retrieval

Jianfeng Dong, Xirong Li, Cees G. M. Snoek

arXiv:1709.01362v3cs.CV

TL;DR

Caption retrieval seeks the best sentence for a given image or video, but existing approaches commonly rely on joint subspaces. This paper maps text into visual feature spaces with Word2VisualVec and reports leading results in the NIST TrecVid challenge, where Word2VideoVec achieved MIR values from 0.097 to 0.110 across two evaluation sets.

  • Problem

    Caption retrieval must identify the sentence that best describes a given image or video, while prevailing methods represent both modalities in a common latent subspace.

    Source: §I · p. 1

  • Method

    Word2VisualVec combines multi-scale sentence vectorization with a multilayer perceptron to predict visual or visual-audio features from text for retrieval.

    Source: §I · p. 2, A. Architecture · p. 4, §V · p. 10

  • Results

    Word2VideoVec led both NIST TrecVid evaluation sets with MIR ranging from 0.097 to 0.110 among 21 submissions from seven teams.

    Source: D. Comparison to the State-of-the-Art · p. 10

  • Takeaways & Limitations

    The results support resolving image and video caption retrieval in a visual feature space exclusively, with text encodings capturing semantic and visual similarities.

    Source: §V · p. 10

Abstract

from arXiv · show

This paper strives to find amidst a set of sentences the one best describing the content of a given image or video. Different from existing works, which rely on a joint subspace for their image and video caption retrieval, we propose to do so in a visual space exclusively. Apart from this conceptual novelty, we contribute \emph{Word2VisualVec}, a deep neural network architecture that learns to predict a visual feature representation from textual input. Example captions are encoded into a textual embedding based on multi-scale sentence vectorization and further transferred into a deep visual feature of choice via a simple multi-layer perceptron. We further generalize Word2VisualVec for video caption retrieval, by predicting from text both 3-D convolutional neural network features as well as a visual-audio representation. Experiments on Flickr8k, Flickr30k, the Microsoft Video Description dataset and the very recent NIST TrecVid challenge for video caption retrieval detail Word2VisualVec's properties, its benefit over textual embeddings, the potential for multimodal query composition and its state-of-the-art results.

I. INTRODUCTION

The paper reframes image and video caption retrieval as matching text and visual instances in a visual feature space, using Word2VisualVec to predict visual features from sentences.

  • Caption retrieval selects the sentence best describing a given image or video from a set of possible sentences.Source: §I · p. 1
  • Unlike prevailing methods that learn a common latent subspace, the paper performs retrieval exclusively in the visual feature space.Source: §I · p. 1
  • Word2VisualVec predicts deep visual features from natural-language descriptions instead of predicting image pixels or class labels.Source: §I · pp. 1–2
  • The sentence encoder combines word2vec and recurrent processing before transforming the representation into a higher-dimensional visual feature space.Source: §I · p. 2
  • The paper contributes a multi-scale sentence vectorization and multi-layer perceptron architecture that can predict several visual features and generalize to visual-audio representations.Source: §I · p. 2

II. RELATED WORK

Prior caption-retrieval systems commonly map image or video and text representations into shared subspaces, whereas this work maps text directly into visual features for retrieval.

  • State-of-the-art video caption retrieval methods commonly encode video and text separately before projecting both into a shared latent subspace.Source: A. Caption Retrieval · p. 2
  • The proposed approach performs image and video caption retrieval directly in the visual space, removing the need to learn a visual-side mapping.Source: A. Caption Retrieval · p. 2
  • BoW sentence representations are limited by vocabularies defined from relatively small multimodal training datasets.Source: B. Sentence Vectorization · p. 2
  • Word2vec supplies a larger text-derived vocabulary, complementing BoW when training-image descriptions omit words for relevant concepts.Source: B. Sentence Vectorization · pp. 2–3
  • Word2VisualVec combines BoW, word2vec, and GRU sentence features early, allowing the model to learn how to combine them.Source: B. Sentence Vectorization · p. 3
  • Because Word2VisualVec predicts visual features from text alone, sentence vectorization can be precomputed for large-scale caption retrieval.Source: B. Sentence Vectorization · p. 3, Figure 2 · p. 3

III. WORD2VISUALVEC

Word2VisualVec maps natural-language descriptions into visual feature space using multi-scale sentence representations and a multilayer perceptron. Its sentence encoding combines BoW, mean-pooled word2vec, and an RNN representation before nonlinear transformation to the visual output.

Visual-space mapping

  • Word2VisualVec learns a mapping from a sentence representation s(q) to a visual feature representation r(q) in the same d-dimensional space as φ(x).The visual feature φ(x) is instantiated with a pretrained ConvNet feature vector.Source: §III · p. 3

Multi-scale sentence vectorization

  • Multi-scale sentence vectorization concatenates BoW, word2vec, and RNN-based text encodings.The resulting vector is used as input to the subsequent multilayer perceptron.Source: A. Architecture · pp. 3–4
  • BoW encodes occurrences from a prespecified vocabulary, while word2vec supplies broader vocabulary coverage through mean pooling over word embeddings.The paper trains a 500-dimensional word2vec model on English tags from 30 million Flickr images, producing a vocabulary of 1.7 million words.Source: A. Architecture · p. 4
  • The RNN addresses the word-order limitation of BoW and word2vec, using its last hidden state h|q| as the sentence representation.The adopted GRU updates hidden states with update and reset gates, using word2vec embeddings to initialize its input embedding matrix.Source: A. Architecture · p. 4

Text transformation via a multilayer perceptron

  • The multilayer perceptron applies affine transformations and ReLU activations through hidden layers until the output reaches the visual feature space.The model parameters include the transformation weights and biases for the hidden layers and output mapping.Source: A. Architecture · p. 4
  • Increasing depth increases learning capacity but also creates more training-loss minima that may generalize poorly to unseen test data.The experiments analyze how deep Word2VisualVec can become without losing generalization ability.Source: A. Architecture · p. 4

B. Learning algorithm

The model learns to reconstruct visual features from relevant image–sentence pairs by minimizing mean squared error. Training uses RMSprop with dropout, learning-rate scheduling, early stopping, and a finite epoch limit.

Objective function

  • The visual-space objective can align different captions that describe similar content, such as different verb tenses or visually similar objects.The paper gives “A dog leaps over a log” and “A dog is leaping over a fallen tree” as an example.Source: B. Learning algorithm · p. 4
  • Word2VisualVec minimizes mean squared error between the predicted sentence feature r(q) and the target visual feature φ(x).The training set contains relevant image–sentence pairs, and the loss is optimized over those pairs.Source: B. Learning algorithm · p. 4

Optimization

  • Optimization uses stochastic gradient descent with RMSprop, whose adaptive scaling is intended to prevent the learning rate from shrinking effectively over time.The reported initial learning rate is η = 0.0001, with decay γ = 0.9 and ϵ = 10^-6.Source: B. Learning algorithm · pp. 4–5
  • Dropout is applied to all hidden layers, the learning rate is halved after three stagnant validation epochs, and early stopping follows ten stagnant epochs.Training is capped at 100 epochs.Source: B. Learning algorithm · p. 5

C. Image Caption Retrieval

For image caption retrieval, test sentences are transformed into visual-space vectors in advance and ranked by proximity to the given image feature. Training requires image–sentence pairs, but test images are not needed to vectorize the sentence pool.

Image caption retrieval

  • Image caption retrieval selects the sentence nearest to a given image in visual feature space.Each test sentence is forward-propagated through Word2VisualVec to obtain r(q), allowing the sentence pool to be vectorized beforehand.Source: C. Image Caption Retrieval · p. 5
  • Image–sentence pairs are required for training Word2VisualVec, but no test image is needed when computing a test sentence’s representation.This separates sentence-pool vectorization from retrieval against the given image.Source: C. Image Caption Retrieval · p. 5

D. Video Caption Retrieval

Word2VisualVec extends to video caption retrieval by projecting sentences directly into video-feature space. The video variant can target visual or visual-audio representations built from frame-level and 3-D ConvNet features.

Video-space mapping

  • For video caption retrieval, sentences are projected into the video feature space rather than a joint subspace.The paper names this video variant Word2VideoVec.Source: D. Video Caption Retrieval · p. 5

Video feature representation

  • Video representations use image ConvNets on individual frames and 3-D ConvNets on consecutive-frame sequences.For the short clips used in the experiments, video features are obtained by mean pooling frame-level feature vectors.Source: D. Video Caption Retrieval · p. 5

Video-space mapping

  • Word2VideoVec transforms an input sentence into either a visual video feature vector or a visual-audio video feature vector.The formulation preserves the same direct text-to-feature-space principle across these target representations.Source: D. Video Caption Retrieval · p. 5

IV. EXPERIMENTS

Experiments evaluate Word2VisualVec across image and video caption retrieval benchmarks, sentence vectorization choices, visual features, model depth, and training-data composition. Multi-scale vectorization and suitable deep visual features perform strongly, while data regime affects the preferred configuration.

Sentence vectorization

  • Multi-scale sentence vectorization outperforms its single-scale counterparts on the evaluated caption-retrieval tasks.The compared representations combine BoW, word2vec, and GRU-based encodings.Source: A. Properties of Word2VisualVec · p. 5
  • Word2vec, BoW, and GRU are suited to different caption types: vocabulary coverage favors word2vec, while narrative captions favor GRU.Word2vec can overweight high-level semantic similarity, making BoW more discriminative in some examples.Source: A. Properties of Word2VisualVec · p. 6

Visual features

Training data

Model depth

  • Three MLP layers peak on Flickr8k and four layers on Flickr30k, while increasing depth also increases overfitting risk.The selected architectures are 500-2048-2048 and 500-2048-2048-2048, respectively, based on validation performance.Source: A. Properties of Word2VisualVec · p. 7

B. Word2VisualVec versus word2vec

Word2VisualVec produces a text representation organized around visual content, bringing captions for the same image closer and captions for different images farther apart than word2vec. However, this comparison is not fully fair because word2vec was not designed for image–text relevance.

Representation comparison

  • Word2VisualVec maps word2vec-based sentence representations into ResNet-152 feature space, where captions for the same image cluster more closely.The representation is intended to capture shared visual semantics despite wording differences across independently written captions.Source: B. Word2VisualVec versus word2vec · p. 7
  • Word2VisualVec better handles wording variance among captions describing the same visual content.The paper illustrates this behavior with examples in which distinct words express shared semantics.Source: B. Word2VisualVec versus word2vec · p. 7
  • Captions for different images are more distant in Word2VisualVec space than in word2vec space.The comparison uses t-SNE visualizations and intra-cluster versus inter-cluster distance histograms on 5K Flickr30k test sentences.Source: B. Word2VisualVec versus word2vec · p. 7, Figure 4 · p. 8

Limitations

  • The models can both fail when descriptions of the same content differ substantially in subjects and objects.One example involves descriptions using teenagers versus people and shirt versus paper.Source: B. Word2VisualVec versus word2vec · pp. 7–8
  • The word2vec comparison is not completely fair because word2vec was not designed to fit image–text relevance.Word2VisualVec explicitly exploits the link between modalities for retrieval.Source: B. Word2VisualVec versus word2vec · p. 8

C. Word2VisualVec for multi-modal querying

Word2VisualVec supports multimodal query composition by adding and subtracting predicted visual features for query words. In an example replacing bicycle with motorbike, retrieval shifts toward motorbikes in street scenes.

Query composition

  • Multimodal queries are composed by subtracting and adding Word2VisualVec-predicted visual features for query words.The composed visual feature is visualized and used to retrieve nearest images and their sentences.Source: C. Word2VisualVec for multi-modal… · p. 8, Figure 5 · p. 9
  • Replacing bicycle with motorbike produces retrieved images emphasizing motorbikes in street scenes.The bicycle feature is subtracted and the motorbike feature is added; motorbike imagery also appears in the corresponding visualization.Source: C. Word2VisualVec for multi-modal… · pp. 8–9

D. Comparison to the State-of-the-Art

Word2VisualVec compares favorably with image-caption retrieval alternatives and leads the reported NIST TrecVid evaluation. Its failures show that retrieval remains constrained by how well visual features represent small objects and actions.

Image caption retrieval

  • The multi-scale model has complexity O(m × s + s × g + (m + s + g) × 2048 + 2048 × d).This is higher than one compared method but lower than another method using time-consuming Fisher vector encoding.Source: D. Comparison to the State-of-the-Art · p. 9

Video caption retrieval

  • The NIST test set contains 1,915 Vine videos, each about 6 seconds long, with 3,830 sentences produced by eight annotators.The sentences are divided into subsets A and B so descriptions of the same video are not in the same subset.Source: D. Comparison to the State-of-the-Art · pp. 9–10
  • Word2VideoVec leads both NIST TrecVid 2016 test subsets, with MIR ranging from 0.097 to 0.110 across the evaluation.The evaluation covers 21 submissions from seven teams, and predicting visual-audio features further improves the results.Source: D. Comparison to the State-of-the-Art · p. 10

Limitations

  • Retrieval errors arise when visual features inadequately represent small objects or actions.Examples include an accordion missed by a ResNet feature and a phone-throwing action missed by an averaged video feature.Source: D. Comparison to the State-of-the-Art · p. 10

E. Limits of caption retrieval and possible extensions

Caption retrieval assumes that at least one relevant sentence exists for every query image or video. For unconstrained content, the paper suggests enlarging the sentence pool or generating novel captions as future extensions.

Scope boundary

  • Caption retrieval assumes that each query image or video has at least one relevant sentence in the candidate pool.The authors state that this assumption may not hold for arbitrary unconstrained content.Source: E. Limits of caption retrieval and… · p. 10

Possible extensions

  • Possible extensions include enlarging the sentence pool or generating new captions from visually relevant phrases.The paper leaves caption construction beyond retrieval for future exploration.Source: E. Limits of caption retrieval and… · p. 10

V. CONCLUSIONS

Word2VisualVec demonstrates that image and video caption retrieval can be resolved exclusively in visual feature space, with strong video retrieval performance further improved by predicting visual-audio features.

  • Word2VisualVec demonstrates the viability of resolving image and video caption retrieval exclusively in a visual feature space.Source: §V · p. 10
  • Word2VisualVec performs well against 19 alternative approaches in the NIST TrecVid 2016 video caption retrieval benchmark.Source: Figure 6 · p. 10
  • Predicting a visual-audio feature can further improve Word2VideoVec’s TrecVid video caption retrieval performance.Source: Figure 6 · p. 10
  • The method transforms natural-language sentences into visual feature representations that capture semantic and visual similarities.Source: §V · p. 10
  • Word2VisualVec supports multimodal query composition by adding or subtracting predicted visual features of specific words from a query image.Source: §V · p. 10

Related papers

Loading 1709.01362v3…