Source-linked AI summary
Explain Images with Multimodal Recurrent Neural Networks
Junhua Mao, Wei Xu, Yi Yang, Jiang Wang, Alan L. Yuille
TL;DR
The paper addresses image description and image–sentence retrieval with a multimodal recurrent neural network. Across three benchmark datasets, the model achieves state-of-the-art performance in sentence generation and both retrieval directions, while its image features remain fixed during training because of limited data.
Problem
Existing retrieval methods can only label query images with sentence annotations from images already present in datasets, motivating novel sentence generation alongside retrieval.
Method
The m-RNN combines a recurrent language model, a convolutional image network, and a multimodal layer that connects their representations.
Results
The m-RNN performs at the state of the art for sentence generation, sentence retrieval given a query image, and image retrieval given a query sentence across three benchmark datasets.
Takeaways & Limitations
The framework supports both generating novel image descriptions and retrieving images or sentences, and can be extended with more complex image features and language models.
Takeaways & Limitations
The authors fix the image features and deep CNN during training because the experimental datasets contain fewer than 30K images.
Abstract
from arXiv · showhide
In this paper, we present a multimodal Recurrent Neural Network (m-RNN) model for generating novel sentence descriptions to explain the content of images. It directly models the probability distribution of generating a word given previous words and the image. Image descriptions are generated by sampling from this distribution. The model consists of two sub-networks: a deep recurrent neural network for sentences and a deep convolutional network for images. These two sub-networks interact with each other in a multimodal layer to form the whole m-RNN model. The effectiveness of our model is validated on three benchmark datasets (IAPR TC-12, Flickr 8K, and Flickr 30K). Our model outperforms the state-of-the-art generative method. In addition, the m-RNN model can be applied to retrieval tasks for retrieving images or sentences, and achieves significant performance improvement over the state-of-the-art methods which directly optimize the ranking objective function for retrieval.
1 Introduction
The paper addresses the limitation that retrieval systems can reuse only existing annotations, proposing m-RNN to generate novel image descriptions while also supporting retrieval.
- Retrieval-based methods label query images with sentences from existing dataset images, limiting descriptions of unseen object-scene combinations.
- The m-RNN jointly targets novel sentence generation and image- or sentence-retrieval tasks.
- The model combines a sentence language model, a deep CNN image component, and a multimodal connection trained with a perplexity-based cost.
- Three benchmark datasets show significant outperformance over state-of-the-art methods for sentence generation and image-and-sentence retrieval using the same image feature networks.
2 Related Work
Prior work includes retrieval models that align image and sentence features in a shared space and generative models that learn multimodal sentence distributions.
- Image-sentence retrieval: Image-sentence retrieval methods extract image and sentence features, then learn a common semantic space through a ranking objective.
- Image-sentence retrieval: Retrieval systems support querying sentences from images or images from sentences by calculating distances between their representations.
- Generating novel sentence descriptions for images: Rule-based generation associates parsed sentence parts with image objects or attributes to produce syntactically correct descriptions.
- Generating novel sentence descriptions for images: Other generative methods learn probability densities over multimodal inputs, producing richer and more flexible sentence structures.
- Generating novel sentence descriptions for images: Unlike a Log-BiLinear model using five words of context, the proposed approach stores temporal context in recurrent layers.
3 Model Architecture
The m-RNN combines recurrent sentence processing with deep image features in a multimodal layer that predicts the next-word distribution across time.
- 3.1 Simple recurrent neural network: A simple RNN uses input-word, recurrent, and output layers, with recurrent connections linking representations across temporal frames.
- 3.2 Our m-RNN model: The m-RNN is deeper than a simple RNN, using an input-word layer, two word-embedding layers, a recurrent layer, a multimodal layer, and a softmax layer.
- 3.2 Our m-RNN model: Word embeddings convert one-hot inputs into 128-dimensional dense representations that reduce parameters and encode semantic relationships.
- 3.2 Our m-RNN model: The model randomly initializes and learns word embeddings, using the second embedding layer as the final word representation.
- 3.2 Our m-RNN model: The recurrent update maps the previous recurrent activation into the word space, adds the current word representation, and applies ReLU.
- 3.2 Our m-RNN model: A 512-dimensional multimodal layer adds transformed word, recurrent, and image features before the softmax predicts the next-word probability distribution.
4 Training the m-RNN
The m-RNN is trained with a perplexity-based cost that evaluates words conditioned on preceding context and corresponding images. Minimizing this differentiable objective maximizes the probability of generating the training sentences.
- Perplexity evaluates a word sequence by the probabilities of each word given its preceding words and the corresponding image.P(wn|w1:n−1, I) is produced by the model’s SoftMax layer.
- The cost function combines the average log-likelihood of training-set words given their context and images with a regularization term.
- N denotes the number of words in the training set, while θ denotes the model parameters.
- Minimizing the cost is equivalent to maximizing the probability of generating the training sentences given their corresponding images.Because the cost is differentiable, model parameters are learned with backpropagation.
5 Learning of Sentence and Image Features
Gradients from the loss function can reach both the language and image components of the m-RNN. However, the experiments keep the CNN features and network fixed because the datasets contain fewer than 30K images.
- Backpropagated loss gradients update the word-embedding and recurrent layers in the language-modeling component.
- The image component uses features from the seventh layer of a pretrained CNN, specifically AlexNet-derived decaf features.These features were also widely used by previous multimodal methods.
- Fewer than 30K images constrain training, so the experiments fix both the image features and the deep CNN network.The authors propose fine-tuning the CNN on larger datasets in future work.
6 Sentence Generation, Image and Sentence Retrieval
The trained m-RNN supports sentence generation plus sentence and image retrieval. It generates words from conditional probabilities and uses perplexity-based affinity, with normalized probability for sentence retrieval.
- The trained model supports sentence generation, sentence retrieval for a given image, and image retrieval for a given sentence.
- Sentence generation begins with a start sign or reference words, then selects subsequent words from P(w|w1:n−1, I).The model samples from the next-word distribution, although maximum-probability selection performs slightly better in practice.
- Image retrieval ranks candidate images by the perplexity of generating the query sentence given each image.Perplexity serves as an affinity measurement between sentences and images.
- Sentence retrieval uses normalized probability P(w1:L|I)/P(w1:L) instead of raw probability or perplexity.This reduces the influence of sentences containing frequently appearing words that may have high probability for any image.
- The marginal image probability is approximated using images sampled from the training set.
7 Experiments
Experiments evaluate m-RNN on three image-caption benchmarks using generation and retrieval metrics. Across datasets, the model improves sentence generation and retrieval performance over baselines and prior methods, while results on IAPR TC-12 also establish reference retrieval scores.
- Experimental setup: The evaluation covers IAPR TC-12, Flickr8K, and Flickr30K, with sentence generation and retrieval tasks assessed using established benchmark protocols.Generation uses perplexity and BLEU; retrieval uses recall-based measures and median rank.
- Evaluation metrics: Perplexity and BLEU measure sentence-generation quality, while R@K and median rank measure sentence and image retrieval.Higher R@K and lower median rank generally indicate better retrieval performance.
- Sentence generation: m-RNN outperforms its RNN baseline on both perplexity and BLEU, and surpasses state-of-the-art methods on perplexity, B-1, and B-3.Its B-2 result is reported as comparable to the state of the art.
- IAPR TC-12 retrieval: m-RNN significantly outperforms three comparison methods on IAPR TC-12 sentence retrieval using a 100-image shortlist.The compared systems include bowdecaf and two multimodal deep models.
- IAPR TC-12 retrieval: 20.9% of top-ranked retrieved images and 13.2% of top-ranked retrieved sentences are groundtruth on IAPR TC-12.These R@K results are reported as reference values because publicly available comparison scores were unavailable.
- Flickr retrieval: On Flickr8K and Flickr30K, m-RNN outperforms state-of-the-art retrieval methods in most evaluation metrics, including comparisons using the same decaf image features.More sophisticated object-detection-based image features improve performance for some methods, but m-RNN remains better in most metrics without them.
8 Conclusion
The m-RNN framework achieves state-of-the-art performance across sentence generation and both image–sentence retrieval directions. It can also incorporate more complex image features and language models.
- The m-RNN framework performs at the state-of-the-art in sentence generation, sentence retrieval given a query image, and image retrieval given a query sentence.
- The framework can be extended with more complex image features, including object detection features, and more sophisticated language models.