PaperlayerView exact revision on arXiv

Source-linked AI summary

Show and Tell: Lessons learned from the 2015 MSCOCO Image Captioning Challenge

Oriol Vinyals, Alexander Toshev, Samy Bengio, Dumitru Erhan

arXiv:1609.06647v1cs.CV

TL;DR

The paper addresses the challenge of generating fluent, semantically appropriate natural-language descriptions from images. It presents NIC, an end-to-end CNN–RNN generative model trained by maximizing sentence likelihood, and reports strong dataset results plus first-place MSCOCO competition performance. The study also identifies evaluation and task-scope limitations that motivate further work.

  • Problem

    Image captioning requires representing objects, relationships, attributes, and activities while expressing them in natural language, making it substantially harder than image classification or object recognition.

  • Method

    NIC uses a pretrained deep CNN to encode an image and an RNN decoder to generate a target description, training the joint model to maximize p(S|I).

  • Results

    NIC ranked first in both automatic and human evaluations in the 2015 MSCOCO challenge, tying another team in human evaluation, and generated diverse captions with top-15 BLEU agreement of 58.

  • Takeaways & Limitations

    The experiments support end-to-end neural image captioning as a robust approach across several datasets, with performance improving as larger image-description datasets become available.

  • Takeaways & Limitations

    The best performance at a relatively small beam size suggests model overfitting or misalignment between likelihood training and human judgment, while image descriptions remain ambiguous and one of many interpretations.

Abstract

from arXiv · show

Automatically describing the content of an image is a fundamental problem in artificial intelligence that connects computer vision and natural language processing. In this paper, we present a generative model based on a deep recurrent architecture that combines recent advances in computer vision and machine translation and that can be used to generate natural sentences describing an image. The model is trained to maximize the likelihood of the target description sentence given the training image. Experiments on several datasets show the accuracy of the model and the fluency of the language it learns solely from image descriptions. Our model is often quite accurate, which we verify both qualitatively and quantitatively. Finally, given the recent surge of interest in this task, a competition was organized in 2015 using the newly released COCO dataset. We describe and analyze the various improvements we applied to our own baseline and show the resulting performance in the competition, which we won ex-aequo with a team from Microsoft Research, and provide an open source implementation in TensorFlow.

1 INTRODUCTION

The paper introduces NIC, a single end-to-end model that uses a CNN to encode images and an RNN to generate natural-language descriptions. It reports strong results across datasets and first-place performance in the 2015 MSCOCO competition, tied in human evaluation.

  • Motivation: Image captioning must represent objects, relationships, attributes, and activities while expressing them in natural English, making it harder than classification or recognition.The task could help visually impaired people understand images on the web.
  • Model and contributions: NIC replaces stitched-together components with one joint model trained to maximize the likelihood p(S|I) of a target word sequence given an image.The model is fully trainable using stochastic gradient descent.
  • Model and contributions: A deep CNN encodes the image into a fixed-length representation that an RNN decoder uses to generate the sentence.The CNN is first pretrained for image classification, and its last hidden layer supplies the RNN input.
  • Results: BLEU 59 versus 25 for the previous state of the art on Pascal, 66 versus 56 on Flickr30k, and 28 versus 19 on SBU were reported for NIC.Human performance on Pascal was reported as 69.
  • Results: NIC placed first in automatic metrics and tied for first with another team in human evaluation at the 2015 MSCOCO competition.The competition followed improvements to the initial model.

2 RELATED WORK

Earlier image-captioning systems commonly relied on hand-designed recognition, retrieval, templates, or other modular constructions. NIC instead combines deep image classification networks with recurrent sequence modeling to generate descriptions directly from images.

  • Earlier systems: Traditional video-description systems combined visual primitive recognizers with structured formal languages and rule-based text generation, limiting them to narrow domains.These systems were described as heavily hand-designed and relatively brittle.
  • Earlier systems: Template and phrase-assembly methods converted detected objects, relationships, or scene-element triplets into descriptions, restricting expressivity.These approaches began with detections and assembled text from predefined components.
  • Retrieval methods: Image-text retrieval methods ranked descriptions near an image in a shared embedding space but did not generate novel descriptions.They generally could not describe previously unseen compositions of objects.
  • Neural generation: NIC combines a convolutional network for image classification with a recurrent network for sequence modeling in a single description-generating network.The RNN is trained within the end-to-end network, with a CNN-processed image replacing the source sentence used in machine translation.
  • Contemporary neural methods: Contemporary neural approaches predicted the next word from an image and prior words, while NIC differed through a more powerful RNN and its visual-input design.The paper identifies feedforward and recurrent alternatives as closely related work.
  • Visual grounding: Other methods explicitly modeled visual anchoring through image-region attention, word-to-region alignment, or visual representations for sentence parts.These approaches sought to connect generated language more explicitly to image content.

3 MODEL

The model generates image descriptions with an end-to-end probabilistic architecture that combines a CNN image encoder with an LSTM-based recurrent language model. It maximizes sentence likelihood by predicting each word from the image and preceding words, using shared parameters across the unrolled LSTM.

  • 3 MODEL: The model directly maximizes the probability of a correct description given an image.The image is denoted I, the sentence S, and θ contains the model parameters.
  • 3 MODEL: A recurrent neural network represents the preceding variable-length word sequence with a fixed-length hidden state updated after each input.The update uses a nonlinear function f of the previous state and current input.
  • 3 MODEL: The recurrent model uses an LSTM, whose gated memory cell controls forgetting, input reading, and output emission.Multiplicative gates address exploding and vanishing gradients; the cell uses forget, input, and output gates.
  • 3 MODEL: A CNN maps the image to an embedding, while word embeddings map caption words into the model’s input space.The CNN is used as an image representation and the words are represented with an embedding model.
  • 3 MODEL: During training, the unrolled LSTM predicts each caption word from the image and preceding words, with all copies sharing parameters.The image is processed once, and the loss sums the negative log likelihood of the correct word at each step.
  • 3 MODEL: At inference, the model generates sentences either by sampling successive words or by retaining multiple high-scoring partial sentences with beam search.Generation ends when the special end-of-sentence token is sampled or a maximum length is reached.

4 EXPERIMENTS

The experiments assess the model’s effectiveness across several metrics, data sources, and architectures, enabling comparison with prior work.

  • 4 EXPERIMENTS: The evaluation varies metrics, data sources, and model architectures to compare the model with prior art.The paper describes the experiments as an extensive assessment of model effectiveness.

4.1 Evaluation Metrics

The paper combines human judgments with automatic metrics to evaluate generated descriptions, while recognizing limitations in metric fidelity and ranking-based evaluation. Human evaluation uses worker ratings, and automatic evaluation includes BLEU, perplexity, CIDER, METEOR, and ROUGE.

  • 4.1 Evaluation Metrics: BLEU measures word n-gram precision against reference sentences and is commonly used in image description evaluation.BLEU-1 evaluates unigrams, whereas BLEU-n averages precision over 1- to n-grams.
  • 4.1 Evaluation Metrics: Perplexity evaluates the geometric mean of inverse probabilities for predicted words and supports model selection and hyperparameter tuning.The paper does not report perplexity because BLEU is preferred.
  • 4.1 Evaluation Metrics: CIDER measures consistency between generated and reference n-grams, weighting occurrences by saliency and rarity.It was introduced for and used by the MS COCO Captioning challenge.
  • 4.1 Evaluation Metrics: METEOR and ROUGE are reported alongside BLEU because the listed metrics have different shortcomings.The paper adds these metrics to broaden automatic evaluation.
  • 4.1 Evaluation Metrics: Ranking-based evaluation can use recall@k but becomes impractical as image complexity and dictionary size make possible sentences grow exponentially.The paper therefore favors evaluation metrics for generation rather than ranking.
  • 4.1 Evaluation Metrics: Human raters score whether descriptions are error-free, mildly erroneous, related, or unrelated, using a four-point scale.Each image is rated by two workers, and disagreement is averaged.

4.2 Datasets

The evaluation uses multiple English image-caption datasets with differing annotation quality and established train/test protocols. Most datasets provide five annotations per image, while SBU contains noisier owner-written descriptions.

  • 4.2 Datasets: The evaluation covers datasets consisting of images paired with English sentences describing their content.The paper introduces dataset statistics before reporting results.
  • 4.2 Datasets: Except for SBU, each image has five relatively visual and unbiased annotations; SBU descriptions are owner-written and noisier.SBU descriptions are not guaranteed to be visual or unbiased because image owners supplied them.
  • 4.2 Datasets: PASCAL is used only for testing after training on another dataset, while SBU holds out 1,000 test images and COCO-4k holds out 4,000 validation images.The remaining SBU images train the model, and COCO-4k is drawn randomly from MSCOCO validation data.

4.3 Results

The results examine generalization, transfer, caption novelty, ranking, and human evaluation, while highlighting overfitting and metric limitations. NIC performed strongly across several datasets and generated diverse captions, but domain mismatch and automated metrics remained important constraints.

  • Training Details: Overfitting was a major training challenge because high-quality datasets contained fewer than 100000 images.Pretraining the CNN weights helped generalization, while initializing word embeddings from a news corpus produced no significant gains.
  • Transfer Learning, Data Size and Label Quality: MSCOCO-to-SBU transfer reduced performance from 28 to 16 because SBU uses weaker labels and a larger, noisier vocabulary.SBU provides more training data, but its captions are not human-generated descriptions.
  • Generation Diversity Discussion: About half of the top 15 generated sentences were completely novel while retaining a similar BLEU score, indicating diverse and sufficiently high-quality descriptions.The best candidate matched a training sentence 80% of the time, whereas the top-15 list produced novel descriptions about half the time.
  • Ranking Results: NIC performed strongly on image-description and description-image ranking tasks, although the paper characterizes ranking as an unsatisfactory evaluation of generation.The ranking results are reported in Tables 4 and 5, covering Flickr8k and Flickr30k.
  • Human Evaluation: Human evaluation placed NIC above a reference system but below groundtruth, revealing that BLEU does not fully capture the difference between generated and human descriptions.The authors therefore report that better evaluation metrics are still needed.

5 THE MS COCO IMAGE CAPTIONING CHAL-

The MS COCO challenge evaluated image-captioning systems on unseen validation and test images, combining server-based submissions with human judging. The authors entered the competition and built improvements on their baseline model.

  • The challenge used MS COCO 2014 training data, validation and test submissions, and a five-attempt limit per group to limit test-set overfitting.Human judges evaluated competing approaches, and winners were invited to present at a CVPR 2015 workshop.
  • Human judges evaluated competing approaches after results were submitted to the evaluation server.
  • The authors entered the competition and explored techniques that extended their baseline image-captioning model.

5.1 Metrics

The competition emphasized CIDER for ranking teams and hyper-parameter selection, while comparing automatic metrics with human rankings. Human rankings varied substantially across metrics, with METEOR aligning best among the reported measures.

  • CIDER was emphasized because competition organizers used it to rank teams and the authors used it for hyper-parameter selection.
  • Automatic metrics correlated strongly with one another, but differed in how highly they ranked humans relative to image-captioning systems.
  • METEOR ranked humans highest among the automatic metrics, placing them third, compared with sixth for CIDER and thirteenth for BLEU.
  • BLEU ranked humans 13th out of 16, whereas CIDER ranked them 6th and METEOR ranked them third.

5.2 Improvements Over Our CVPR15 Model

The authors improved their CVPR 2015 model through image-model upgrades and fine-tuning, training changes, ensembles, and beam-size selection. Fine-tuning and ensembles added BLEU-4 gains, while a small beam produced the strongest CIDER improvement and increased novelty.

  • Image Model Improvement: Batch Normalization improved MS COCO captioning performance by 2 absolute BLEU-4 points over the earlier GoogleLeNet-based image model.The Batch Normalization model also reduced ImageNet top-5 error from 6.67% to 4.8%.
  • Image Model Fine Tuning: Fine-tuning the CNN after stabilizing the LSTM improved captioning performance by transferring image information unavailable from the fixed pretrained model.Joint training from the beginning corrupted the CNN, so the authors first froze it and later trained both components jointly.
  • Image Model Fine Tuning: 1 BLEU-4 point came from fine-tuning the image model, which enabled captions to predict image-specific colors such as blue and yellow.
  • Ensembles: An ensemble of five Scheduled Sampling models and ten fine-tuned models improved results by a further 1.5 BLEU-4 points.
  • Beam Size Reduction: A beam size of 3 performed best under CIDER, and reducing beam size increased caption novelty by lowering training-caption repetition from 80% to 60%.
  • Beam Size Reduction: The best small beam size suggested either model overfitting or misalignment between likelihood optimization and human judgment.
  • Beam Size Reduction: Reducing beam size produced the largest CIDER improvement and yielded more than 2 BLEU-4 points.

5.3 Competition Results

Competition evaluation combined automatic metrics with human assessment on unseen test images. The authors reported an 8-point absolute BLEU-4 improvement over their submitted paper and showed improved final-model captions in qualitative examples.

  • Automatic Evaluation: Teams could submit up to 5 times on a large, unseen test set, while automatic metrics correlated strongly with caption quality but did not fully characterize it.
  • Automatic Evaluation: The authors improved their BLEU-4 score by 8 points absolute after applying the reported model improvements.
  • Human Evaluation: The 15 most promising submissions and a human baseline were evaluated using five metrics covering human preference, Turing-test performance, correctness, detail, and similarity.
  • Human Evaluation: M1 and M2 determined the competition winner, while the remaining metrics were reported as experimental measures.
  • Qualitative Evaluation: In a random sample of 20 development images, every final-model caption except one was better than the original-model caption.

6 CONCLUSION

NIC generates English image descriptions through an end-to-end neural system and performs strongly across qualitative and quantitative evaluations. The authors view current captioning results as an early stage, with future work targeting more controlled descriptions and application-oriented evaluation.

  • NIC encodes an image with a convolutional neural network and generates a plain-English description with a recurrent network.
  • The model performs strongly on several datasets and ranked first in the 2015 MS COCO challenge under both automatic and human evaluations.
  • Future work should support targeted descriptions grounded in specified image properties or locations, user questions, or tasks, alongside improved evaluation metrics.The authors also suggest evaluating captioning through higher-level goals in applications such as robotics.
Loading 1609.06647v1…