Source-linked AI summary
Language Models for Image Captioning: The Quirks and What Works
Jacob Devlin, Hao Cheng, Hao Fang, Saurabh Gupta, Li Deng, Xiaodong He, Geoffrey Zweig, Margaret Mitchell
TL;DR
Image captioning research lacked a controlled comparison between two leading language-modeling approaches. This paper uses the same CNN to compare them and combines their components, achieving a record COCO BLEU result, while showing that automatic gains do not consistently improve human judgments.
Problem
The paper addresses the need to compare CNN-word-detection-plus-ME and CNN-activation-conditioned RNN captioning approaches under the same visual input.
Method
The authors use a common state-of-the-art CNN to compare ME-based, RNN-based, retrieval, and combined captioning models, including a k-nearest-neighbor consensus method.
Results
1.6 BLEU points advance the best previously published COCO result, but the improvement does not translate to better human quality judgments.
Takeaways & Limitations
ME-based methods generate relatively novel captions and perform better on novel images and human judgments, whereas MRNN achieves stronger BLEU but tends to reproduce training captions.
Takeaways & Limitations
The authors identify generic captions, limited dataset diversity, and caption duplication—especially for MRNN—as unresolved evaluation and modeling concerns.
Abstract
from arXiv · showhide
Two recent approaches have achieved state-of-the-art results in image captioning. The first uses a pipelined process where a set of candidate words is generated by a convolutional neural network (CNN) trained on images, and then a maximum entropy (ME) language model is used to arrange these words into a coherent sentence. The second uses the penultimate activation layer of the CNN as input to a recurrent neural network (RNN) that then generates the caption sequence. In this paper, we compare the merits of these different language modeling approaches for the first time by using the same state-of-the-art CNN as input. We examine issues in the different approaches, including linguistic irregularities, caption repetition, and data set overlap. By combining key aspects of the ME and RNN methods, we achieve a new record performance over previously published results on the benchmark COCO dataset. However, the gains we see in BLEU do not translate to human judgments.
1 Introduction
The paper compares two image-captioning paradigms using a common CNN: a two-stage ME language model and an RNN conditioned directly on CNN representations. It finds distinct trade-offs between BLEU performance, caption novelty, and human judgments.
- The two-stage approach predicts likely caption words with a CNN, then uses an ME language model to generate a sentence covering detected words.
- The alternative feeds final-layer CNN activations directly into an RNN language model to generate the caption sequence.
- Using the same state-of-the-art CNN, the paper empirically compares the strengths and weaknesses of the two language-modeling components.
- The MRNN outperforms the ME language model on BLEU for the COCO dataset, while the ME language model produces more novel captions and handles unmatched images better.
- A simple k-nearest-neighbor method performs near state-of-the-art on this task and dataset.
3. We demonstrate that a state-of-the-art
The paper examines model behavior and evaluation beyond headline automatic scores, including caption reconstruction, linguistic irregularities, human assessment, and dataset issues. These analyses expose differences between retrieval, ME-based, and recurrent approaches.
- A gated recurrent neural network is used in the paper and is described as similar to an LSTM.
- The MRNN-based approach tends to reconstruct captions previously seen in training, whereas the two-stage ME approach produces relatively novel captions with similar or better performance.
- The paper reports that its methods advance state-of-the-art BLEU scores on the COCO dataset.
- Human evaluation is performed on systems with the best performance according to automatic metrics.
- The analysis explores linguistic irregularities, caption repetition, and overlap in the underlying COCO dataset.
2 Models
The paper compares captioning models conditioned on CNN-derived information, including detected words, continuous CNN activations, and image retrieval. It also develops a consensus-caption variant of nearest-neighbor retrieval.
- All compared language models use output from the same state-of-the-art VGGNet CNN, finetuned on Microsoft COCO.The CNN is the 16-layer VGGNet variant, initially trained for ILSVRC2014 classification and then finetuned on COCO.
- 2.1 Detector Conditioned Models: Detector-conditioned models use CNN-predicted words and beam search to generate sentences covering a subset of those words.The detector-conditioned LSTM dynamically removes words after they are mentioned and encodes remaining attributes as an auxiliary continuous feature.
- 2.1 Detector Conditioned Models: The D-ME system analyzes 500-best maximum-entropy outputs and can rerank them using a Deep Multimodal Similarity Model.The DMSM projects image pixels and caption text into a shared vector space and scores their similarity.
- 2.2 CNN Activation Conditioned Models: The MRNN conditions generation directly on the CNN's 4096-dimensional fc7 activation, mapped to a 500-dimensional initial recurrent state.A gated recurrent neural network then generates the caption one word at a time using the previous word and recurrent state, with beam-search decoding.
- 2.3 k-Nearest Neighbor Model: The retrieval baseline selects k = 90 nearest training images and chooses a consensus caption with the highest mean n-gram overlap among m = 125 similar captions.The values of k and m were selected by grid search on the validation set; the consensus caption is intended to describe multiple similar images rather than only the nearest one.
3 Experimental Results
Experiments on COCO compare captioning models using automatic metrics and human judgments, including re-ranked combinations of ME, MRNN, and DMSM systems. Results show that automatic BLEU and perplexity rankings do not consistently match human evaluations.
- The experiments use Microsoft COCO, with 82,783 training images and separate validation and testval splits, each image having five human captions.
- The MRNN has the lowest PPLX and highest BLEU among the language models, improving BLEU by 2.1 absolutely over the D-ME LM baseline.
- The k-nearest neighbor method achieves a higher BLEU score than all other models, although generated captions receive better human-quality judgments than nearest-neighbor captions.
- Combining D-ME, DMSM, and MRNN yields a 1.6 BLEU improvement over D-ME+DMSM after re-ranking hypotheses.
- Human evaluations show D-ME+DMSM outperforming MRNN by 5 percentage points on “Better Or Equal to Human,” while the combined model does not exceed D-ME+DMSM despite higher BLEU.
- The authors cannot pinpoint the exact reason for the mismatch between automated scores and human evaluation.
4 Language Analysis
The two language-modeling approaches exhibit different error patterns: D-ME struggles with anaphora, while MRNN can repeat attributes and training captions. Caption repetition affects all systems, especially MRNN.
- Linguistic irregularities: D-ME has difficulty with anaphora, particularly in phrases such as “on top of it,” whereas MRNN tends to generate these relationships correctly.The contrast is attributed to D-ME maintaining a local context window.
- Linguistic irregularities: MRNN sometimes emits the same attribute more than once, especially when coordination with “and” is present.D-ME instead maintains an explicit coverage state vector tracking emitted attributes.
- Caption repetition: All models produce many captions seen during training and repeat captions across different test images.The paper reports this pattern in Table 6.
- Caption repetition: Generic captions such as “a close up of a plate of food” can apply to many publicly available images.The authors identify this generic-caption behavior as one possible cause of repetition.
5 Image Diversity
The paper links strong retrieval performance and repeated captions to limited diversity in the COCO data. On visually novel images, D-ME+DMSM outperforms retrieval and MRNN by BLEU, while MRNN performs better on highly overlapping images.
- Image diversity: Strong k-nearest-neighbor performance and repeated captions suggest limited diversity in the training and test data.The paper notes that this may partly reflect how COCO was constructed around 80 predefined object categories.
- Image diversity: The authors motivate evaluation on compositionally novel images, whose individual components may appear in training without the full composition appearing.They bin test images by visual overlap with training data using fc7 cosine similarity.
- Results by overlap: D-ME+DMSM exceeds MRNN by 2.1 BLEU on the “20% Least” set but trails it by 2.1 BLEU on the “20% Most” set.The paper interprets this pattern as evidence that D-ME+DMSM generalizes better to novel images than MRNN.
6 Conclusion
The conclusion finds that MRNN achieves stronger BLEU than several individual baselines, while ME LM+DMSM receives better human quality judgments. Combining MRNN, DMSM, and ME-based reranking improves COCO BLEU, but not human judgments.
- Conclusion: MRNN achieves better BLEU than ME LM and LSTM models conditioned on discrete activations, while matching the BLEU of ME LM combined with DMSM.A k-nearest-neighbor retrieval algorithm performs similarly to MRNN on both automatic metrics and human judgments.
- Conclusion: ME LM+DMSM significantly outperforms MRNN in human quality judgments, which the authors hypothesize is partly due to MRNN’s less novel captions.The retrieval baseline also performs similarly to MRNN on human judgments.
- Conclusion: 1.6 BLEU points improve on the best previously published COCO result when MRNN and DMSM scores rerank ME LM n-best outputs with MERT.The improvement in BLEU does not translate into improved human quality judgments.