Source-linked AI summary
Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN)
Junhua Mao, Wei Xu, Yi Yang, Jiang Wang, Zhiheng Huang, Alan Yuille
TL;DR
Image-captioning retrieval methods cannot generate novel descriptions for unseen object–scene combinations. The paper proposes m-RNN, combining deep recurrent language and convolutional vision networks through a multimodal layer, and reports state-of-the-art results across captioning and retrieval tasks.
Problem
Retrieval-based captioning cannot generate novel sentences or describe images containing novel combinations of objects and scenes.
Method
m-RNN combines a deep recurrent language model and deep CNN image representation through a multimodal layer that models next-word probabilities.
Results
m-RNN achieves state-of-the-art performance across sentence generation, image retrieval, and sentence retrieval on four benchmark datasets.
Takeaways & Limitations
The framework connects images and sentences while remaining extensible to more complex image representations and language models.
Takeaways & Limitations
Image features and the deep CNN are fixed during training because of limited data, leaving CNN fine-tuning for future work on larger datasets.
Abstract
from arXiv · showhide
In this paper, we present a multimodal Recurrent Neural Network (m-RNN) model for generating novel image captions. It directly models the probability distribution of generating a word given previous words and an image. Image captions are generated by sampling from this distribution. The model consists of two sub-networks: a deep recurrent neural network for sentences and a deep convolutional network for images. These two sub-networks interact with each other in a multimodal layer to form the whole m-RNN model. The effectiveness of our model is validated on four benchmark datasets (IAPR TC-12, Flickr 8K, Flickr 30K and MS COCO). Our model outperforms the state-of-the-art methods. In addition, we apply the m-RNN model to retrieval tasks for retrieving images or sentences, and achieves significant performance improvement over the state-of-the-art methods which directly optimize the ranking objective function for retrieval. The project page of this work is: www.stat.ucla.edu/~junhua.mao/m-RNN.html .
1 INTRODUCTION
The paper addresses the limitations of caption-retrieval methods by proposing m-RNN for generating novel image descriptions and supporting image–sentence retrieval. It combines language, vision, and multimodal components and reports state-of-the-art performance across three tasks.
- Retrieval-based captioning cannot generate novel sentences or describe novel combinations of objects and scenes.
- The m-RNN model addresses novel image-caption generation, image retrieval, and sentence retrieval.
- The model combines a language model, deep CNN vision component, and multimodal representation trained with a log-likelihood cost.
- Across IAPR TC-12, Flickr 8K, Flickr 30K, and MS COCO, m-RNN significantly outperforms other methods on all three tasks.
2 RELATED WORK
Prior work approaches image description through grammar-based generation, caption recombination, multimodal density modeling, or shared image–sentence embeddings. The paper distinguishes m-RNN through learned word embeddings and repeated direct visual input to the multimodal layer.
- Deep CNNs provide image representations used in modern computer-vision methods and are incorporated into the m-RNN framework.
- Image–sentence retrieval methods learn a shared semantic embedding by optimizing ranking objectives over sentence and image features.
- Image-description methods include grammar-based generation, caption recombination, and probabilistic modeling of multimodal inputs.
- m-RNN differs by learning a two-layer word embedding system rather than relying on a single-layer embedding.
- m-RNN inputs image representations alongside every sentence word instead of storing visual information in the recurrent layer.
3 MODEL ARCHITECTURE
The architecture combines recurrent language processing with deep visual features through a multimodal layer that predicts the next word. It uses learned word embeddings, ReLU recurrence, and direct image input, while fixed CNN features constrain training.
- Simple RNN: A simple RNN uses input-word, recurrent, and output layers, with the output producing the next-word distribution.
- m-RNN architecture: The m-RNN has two word-embedding layers, a recurrent layer, a multimodal layer, and a softmax layer, with shared weights across time frames.
- Word embeddings: The embedding layers learn dense representations encoding syntactic and semantic word information from training data.
- Recurrent layer: The recurrent layer maps the previous recurrent activation into the word-vector space and adds it element-wise to the current word representation.
- Recurrent layer: ReLU recurrence avoids early truncation of backpropagation through time and is described as faster and less prone to saturation or overfitting than sigmoid recurrence.
- Multimodal layer: The multimodal layer adds mapped word-embedding, recurrent, and image representations to connect language and vision.
4 TRAINING THE M-RNN
The m-RNN is trained with a log-likelihood cost related to sentence perplexity given images. Backpropagation minimizes this differentiable objective to maximize training-sentence probabilities.
- Perplexity measures the model’s language-model performance for a sentence conditioned on its corresponding image.For a word sequence w1:L, PPL(w1:L|I) summarizes the probabilities assigned to its words given the image and preceding words.
- The cost function averages word log-likelihoods over training sentences and adds a regularization term.Ns and N denote the numbers of sentences and words, Li is sentence length, and θ represents model parameters.
- The training objective minimizes this differentiable cost, equivalently maximizing the probability of generating the training sentences.Backpropagation learns the model parameters.
5 SENTENCE GENERATION, IMAGE RETRIEVAL AND SENTENCE RETRIEVAL
The trained m-RNN supports sentence generation and bidirectional image–sentence retrieval. It generates words from conditional probabilities and uses sentence likelihood or normalized probability as retrieval affinity.
- The trained m-RNN performs sentence generation, image retrieval, and sentence retrieval.Image retrieval finds relevant images for a sentence, while sentence retrieval finds relevant sentences for an image.
- Sentence generation samples each next word from P(wn|w1:n−1, I), beginning with a start sign or reference words.Selecting the maximum-probability word performs slightly better in practice than sampling.
- Sentence likelihood P(w1:L|I), computed as the product of conditional word probabilities, serves as an affinity measure between images and sentences.This score supports ranking for retrieval.
- Image retrieval ranks dataset images by the probability of generating the query sentence given each image.The highest-ranked images are retrieved, equivalent to perplexity-based image retrieval.
- Sentence retrieval uses normalized probability with the sentence’s marginal probability as the normalization factor.This addresses sentences that receive high probability for many image queries.
6 LEARNING OF SENTENCE AND IMAGE FEATURES
Gradients can reach both the language and vision components, but the experiments keep the pretrained CNN fixed because of limited data. The authors identify larger datasets and CNN fine-tuning as future directions.
- Gradients from the loss backpropagate through the word embeddings, recurrent layer, and vision component.The architecture permits joint updates across language modeling and image representation parts.
- The vision component uses pretrained AlexNet or VggNet features from ImageNet.The paper also notes that RCNN object-detection features combined with AlexNet can outperform whole-image features.
- Limited data required fixing the image features and deep CNN during training in this paper.The authors plan to use larger datasets and fine-tune the CNN parameters in future work.
- Sentence generation takes 25 ms on average on a single-core CPU, excluding image feature extraction.
7 EXPERIMENTS
The experiments evaluate m-RNN on four benchmark datasets using caption-generation and retrieval metrics. Across these datasets, m-RNN generally outperforms baseline and state-of-the-art methods, while stronger image representations and direct visual input improve performance.
- Datasets: Four benchmark datasets— IAPR TC-12, Flickr 8K, Flickr 30K, and MS COCO—provide sentence-level annotations for evaluation.The experiments use standard dataset partitions or selected validation images for testing.
- Evaluation metrics: Sentence generation is evaluated with perplexity and BLEU scores, while retrieval uses R@K and median rank.R@K measures recall among the top K candidates, whereas lower median rank indicates better retrieval.
- IAPR TC-12: m-RNN performs much better than the baseline RNN and state-of-the-art methods on both perplexity and BLEU for IAPR TC-12.The baseline RNN lacks the image representation input and can achieve low perplexity while producing image-inconsistent captions.
- IAPR TC-12: 20.9% of top-ranked retrieved sentences and 13.2% of top-ranked retrieved images are groundtruth on IAPR TC-12.These R@K results are reported for future comparison because publicly available results were unavailable for this dataset.
- Flickr8K: On Flickr8K, m-RNN outperforms competing methods by a large margin with the same image representation and remains better in almost all metrics without object detection.The comparison includes AlexNet-based representations and methods using RCNN-derived features.
- Flickr30K and MS COCO: On Flickr30K and MS COCO, VggNet-based m-RNN outperforms recent state-of-the-art methods in almost all evaluation metrics, with 71% of MS COCO captions novel.VggNet performs better than AlexNet, and the model uses a relatively small recurrent layer.
- Component analysis: A two-layer word embedding system and direct visual input to the multimodal layer improve performance, with the latter increasing B-1 by about 5%.The component comparisons are reported in supplementary experiments.
8 NEAREST NEIGHBOR AS REFERENCE
The m-RNN generates caption hypotheses, retrieves visually similar training images, and reranks the hypotheses by consensus with neighboring captions. Refined image features and consensus reranking improve caption relevance, while oracle results indicate remaining headroom.
- Method: The method generates n hypotheses with beam search, retrieves nearest training-image neighbors, and reranks hypotheses using consensus with their captions.The experiments set n = 10 for reranking and compare hypotheses against captions from neighboring images.
- Nearest-neighbor features: The refined m-RNN image features capture richer and more accurate visual information than original VggNet features for nearest-neighbor retrieval.In one example, original VggNet features fail to retrieve images containing bananas when the target image shows an old woman with bananas.
- Consensus reranking: Consensus scores average similarity to the m nearest neighboring captions, using sentence-level BLEU or CIDEr as the similarity measure.The optimal neighborhood settings reported are k = 60 and m = 175 for BLEU, and k = 60 and m = 125 for CIDEr.
- Results: 3.5 points on validation and 3.3 points on MS COCO test 2014 were gained in BLEU4 through BLEU-based consensus reranking.The corresponding CIDEr improvements were 9.4 points on validation and 9.8 points on the MS COCO test 2014 set.
- Results: Reranking improves the rank of good captions among similar hypotheses that vary in detail and correctness.The paper illustrates the rank changes before and after reranking for ten hypotheses.
- Limitations and headroom: Oracle reranking is surprisingly high, indicating remaining room to improve both the m-RNN model and the reranking strategy.Oracle performance is defined as the upper bound obtained by reranking hypotheses against ground-truth captions.
9 CONCLUSION
The conclusion presents m-RNN as a multimodal framework for caption generation and bidirectional image–sentence retrieval. Its deep recurrent and convolutional components interact through a multimodal layer, supporting flexible image and language representations.
- Conclusion: The m-RNN framework performs at the state-of-the-art in sentence generation, sentence retrieval given an image, and image retrieval given a sentence.The conclusion lists all three tasks as targets of the framework.
- Examples: The conclusion includes generated caption hypotheses for people near a train, a black bear, and a piece of cake.The listed hypotheses show multiple visually similar caption alternatives for each example.
- Reranking: Consensus reranking changes the ordering of caption hypotheses, as illustrated by the original and after-reranking lists.The figure materials provide paired rankings for the example hypotheses.
- Architecture: The model combines a deep RNN, a deep CNN, and a multimodal layer that connects image and sentence representations.The multimodal layer is the interaction point between the two sub-networks.
10 SUPPLEMENTARY MATERIAL
The supplementary material examines model components, retrieval evaluation, BLEU calculation, and retrieval-accuracy curves. It reports that the original architecture performs best among tested variants and outperforms comparison methods on the displayed retrieval task.
- Model variants: Table 9 compares m-RNN variants using VggNet image representations on Flickr30K.The table is paired with Figure 5, which illustrates the seven model variants.
- Word embedding system: Two word embedding layers and their connection to the multimodal layer provide useful information for predicting the next word distribution.The ablation variants remove or replace these embedding components to test their contribution.
- Word embedding system: The original m-RNN performs best among the tested embedding variants, verifying the effectiveness of its two word embedding layers.This result is reported in the Flickr30K comparison.
- Vision–language connection: Directly inputting visual information to the multimodal layer is more effective than the tested variants that input it through the recurrent pathway.The compared variants include models that feed image representations to word embedding layer II, or to both locations.
- Training caveat: The tested recurrent-pathway variants are harder to train and require a very small learning rate to avoid exploding gradients.The paper suggests increasing recurrent-layer dimension or replacing the RNN with an LSTM as future directions.
- Retrieval comparisons: The retrieval curves compare a bag-of-words baseline and multimodal deep models, with the m-RNN significantly outperforming all three comparison methods.The curves report recall against the percentage of retrieved images or sentences.