Source-linked AI summary
Show and Tell: A Neural Image Caption Generator
Oriol Vinyals, Alexander Toshev, Samy Bengio, Dumitru Erhan
TL;DR
Generating fluent image descriptions requires connecting visual content, object relationships, and natural language. NIC addresses this with a joint CNN–RNN model and improves benchmark performance, including BLEU 59 versus 25 on Pascal.
Problem
Automatically generating fluent English descriptions must capture image objects, relationships, attributes, and activities while expressing them in natural language.
Method
NIC uses a convolutional neural network to encode an image and a recurrent neural network to generate a sentence by maximizing its likelihood given the image.
Results
BLEU-1 improved from 25 to 59 on Pascal, from 56 to 66 on Flickr30k, and from 19 to 28 on SBU.
Takeaways & Limitations
NIC generates reasonable English captions across several datasets, with performance expected to increase as image-description datasets grow.
Takeaways & Limitations
The supervised approach faces overfitting because high-quality image-description datasets contain fewer than 100000 images.
Abstract
from arXiv · showhide
Automatically describing the content of an image is a fundamental problem in artificial intelligence that connects computer vision and natural language processing. In this paper, we present a generative model based on a deep recurrent architecture that combines recent advances in computer vision and machine translation and that can be used to generate natural sentences describing an image. The model is trained to maximize the likelihood of the target description sentence given the training image. Experiments on several datasets show the accuracy of the model and the fluency of the language it learns solely from image descriptions. Our model is often quite accurate, which we verify both qualitatively and quantitatively. For instance, while the current state-of-the-art BLEU-1 score (the higher the better) on the Pascal dataset is 25, our approach yields 59, to be compared to human performance around 69. We also show BLEU-1 score improvements on Flickr30k, from 56 to 66, and on SBU, from 19 to 28. Lastly, on the newly released COCO dataset, we achieve a BLEU-4 of 27.7, which is the current state-of-the-art.
1. Introduction
The paper frames automatic image description as a difficult but impactful task and proposes NIC, an end-to-end neural model that generates English sentences from images. NIC combines a pretrained vision CNN with an RNN language decoder and is trained jointly to maximize the likelihood of target descriptions.
- Problem: Automatic image description is challenging because it must produce properly formed English sentences, yet it could help visually impaired people understand images online.The task is described as substantially harder than image classification or object recognition.
- Approach: NIC replaces stitched-together solutions with a single joint model that takes an image I and maximizes p(S|I) for a target word sequence S.The target sequence is drawn from a given dictionary and should describe the image adequately.
- Architecture: NIC uses a deep CNN as an image encoder and an RNN decoder that generates sentences from the CNN’s fixed-length image representation.The CNN is pretrained for image classification, with its last hidden layer provided as input to the RNN decoder.
- Contributions: The system is fully trainable end-to-end with stochastic gradient descent and combines state-of-the-art vision and language sub-networks.The sub-networks can be pretrained on larger corpora to exploit additional data.
2. Related Work
Prior image-description systems relied on hand-designed recognition and language components, retrieval, or limited-composition generation. This work instead combines convolutional and recurrent networks in a single end-to-end model that generates image descriptions.
- Earlier description systems: Earlier video-description systems combined visual primitive recognizers, structured formal languages, and rule-based natural-language conversion, but were brittle and limited to domains such as traffic scenes.These systems were heavily hand-designed.
- Earlier description systems: Still-image systems used object, attribute, and location detections to drive templates or phrase assembly, limiting their expressivity.Farhadi et al. inferred scene-element triplets, while Li et al. assembled descriptions from detected phrases.
- Retrieval-based methods: Image-text retrieval methods ranked descriptions near images in a shared embedding space, including neural co-embeddings of images and sentences or image crops and subsentences.These approaches did not attempt to generate descriptions.
- Limitations: These prior approaches generally could not describe previously unseen object compositions and avoided evaluating the quality of generated descriptions.Individual objects could still have been observed during training.
- Proposed approach: The proposed model combines a convolutional image-classification network with a recurrent sequence-modeling network in one end-to-end system that generates image descriptions.It adapts sequence-generation ideas from machine translation by providing an image processed through a convolutional network instead of a sentence.
3. Model
The model generates image descriptions by directly maximizing their conditional probability, using an RNN with LSTM memory and a CNN image representation. Training predicts each sentence word from the image and preceding words, while inference uses sampling or beam search.
- Model objective: The framework directly maximizes the probability of the correct description given an image.The parameters are optimized over image–sentence training pairs using stochastic gradient descent.
- Recurrent architecture: An RNN models each word conditioned on the image and preceding words through a fixed-length hidden state.The hidden memory h_t is updated after each new input with a nonlinear function.
- Neural components: The recurrent function is implemented with an LSTM, while a CNN provides the image representation.The LSTM addresses vanishing and exploding gradients, and the CNN is used for image recognition and detection.
- Input representation: The unrolled LSTM shares parameters across time and maps the image and words into a common space.The image is input only once; words use one-hot vectors and word embeddings, with special start and stop tokens marking sentence boundaries.
- Inference: Inference generates sentences either by sampling successive words or by retaining the k best partial sentences through beam search.Generation stops at the end-of-sentence token or a maximum length.
4. Experiments
The experiments evaluate the model’s effectiveness across multiple metrics, data sources, and model architectures, enabling comparison with prior work.
- The study conducts an extensive set of experiments to assess model effectiveness.
- The evaluation uses several metrics and data sources.
- The experiments examine multiple model architectures and compare results to prior art.
4.1. Evaluation Metrics
The paper evaluates image descriptions using subjective human ratings and automatic metrics, especially BLEU, while using perplexity for model selection. It favors generation-oriented evaluation over ranking because ranking becomes increasingly unsatisfactory as image and vocabulary complexity grow.
- Human evaluation: Subjective human ratings are treated as the most reliable, though time-consuming, evaluation of a description’s usefulness given an image.The paper uses these ratings to assess whether automatic metrics correlate with human judgments.
- Automatic metrics: BLEU measures word n-gram precision between generated and reference sentences and is the most commonly used automatic metric in image description.The paper acknowledges BLEU’s drawbacks but reports that it correlates well with human evaluations.
- Automatic metrics: Perplexity is used for model selection and hyperparameter tuning on a held-out set, but it is not reported because BLEU is preferred.Perplexity is the geometric mean of the inverse probability assigned to each predicted word.
- Generation versus ranking: The paper prioritizes generation-task metrics over ranking metrics such as recall@k because ranking becomes unsatisfactory as image complexity and dictionary size increase.Ranking transforms sentence generation into a task over an exponentially growing number of possible sentences.
4.2. Datasets
The evaluation uses image–English sentence datasets, with annotation practices and test splits varying across Pascal, SBU, and MSCOCO.
- Dataset composition: Evaluation uses datasets consisting of images paired with English sentences describing those images.
- Dataset composition: Except for SBU, each image has five relatively visual and unbiased annotations; SBU uses owner-provided Flickr descriptions, introducing more noise.
- Evaluation splits: Pascal is used only for testing after training on another dataset, while SBU holds out 1000 images and MSCOCO reserves 4K validation images as COCO-4k.
4.3. Results
Experiments across five datasets show strong captioning performance, while also revealing sensitivity to training data, domain mismatch, and evaluation methodology. The model generates diverse captions and performs well on ranking tasks, but human raters distinguish it from ground-truth descriptions more clearly than BLEU does.
- Main results: 27.2 BLEU-4 was achieved on the official test set, where labels were available only through the official website.The paper argues that BLEU-4 is the more meaningful standard as the field advances.
- Transfer learning and data size: 4 BLEU points better results were obtained by training on Flickr30k than on Flickr8k, which has about four times less training data.The authors attribute this gain to the data-driven, overfitting-prone nature of the process.
- Transfer learning and data size: 53 BLEU-1 was obtained when transferring from Flickr30k to PASCAL, compared with 59 from MSCOCO.PASCAL has no official training set and was collected independently of Flickr and MSCOCO.
- Transfer learning and data size: 28 down to 16 was the performance degradation when the MSCOCO model was run on SBU, whose captions provide weaker labels and a larger, noisier vocabulary.SBU nevertheless offers substantially more training data.
- Caption generation: 58 was the BLEU agreement among the top 15 generated sentences, whose samples were diverse and could show different aspects of the same image.The examples came from beam-search N-best lists rather than only the best hypothesis.
- Evaluation: NIC performed surprisingly well on both ranking descriptions given images and ranking images given descriptions, despite ranking being considered an unsatisfactory evaluation method.Human evaluation found NIC better than a reference system but clearly worse than ground-truth descriptions, indicating BLEU does not fully capture this distinction.
5. Conclusion
The paper presents NIC, an end-to-end neural network that generates reasonable English descriptions from images. NIC encodes images with a convolutional neural network and generates sentences with a recurrent neural network, with robustness shown through qualitative and quantitative evaluations.
- NIC is an end-to-end neural network system that automatically views an image and generates a reasonable description in plain English.
- NIC encodes an image into a compact representation with a convolutional neural network, then generates a corresponding sentence with a recurrent neural network.
- The model is trained to maximize the likelihood of the sentence given the image and evaluated qualitatively and quantitatively across several datasets.Quantitative evaluations use either ranking metrics or [the supplied passage ends here].