Source-linked AI summary
Microsoft COCO Captions: Data Collection and Evaluation Server
Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollar, C. Lawrence Zitnick
TL;DR
Image captioning lacks a consistently evaluated, large-scale human-caption resource. The paper constructs the Microsoft COCO Caption dataset and an evaluation server using multiple metrics, while noting that automatic metrics do not always match human judgment and that BLEU performs poorly for individual sentences.
Problem
Image captioning is a long-standing challenging problem, and comparing systems is difficult because evaluation metrics and their implementations vary.
Method
The paper collects human-generated captions for MS COCO images and provides an evaluation server that scores candidate captions with multiple automatic metrics.
Results
The dataset contains 1,026,459 collected captions, including five-reference and forty-reference caption sets, and the server reports BLEU, METEOR, ROUGE, and CIDEr scores.
Takeaways & Limitations
The dataset and server provide a shared resource and evaluation protocol for comparing image caption generation approaches.
Takeaways & Limitations
Automatic metrics do not always correspond to human judgment, motivating human evaluations to assess caption quality and metric correlation.
Abstract
from arXiv · showhide
In this paper we describe the Microsoft COCO Caption dataset and evaluation server. When completed, the dataset will contain over one and a half million captions describing over 330,000 images. For the training and validation images, five independent human generated captions will be provided. To ensure consistency in evaluation of automatic caption generation algorithms, an evaluation server is used. The evaluation server receives candidate captions and scores them using several popular metrics, including BLEU, METEOR, ROUGE and CIDEr. Instructions for using the evaluation server are provided.
1 INTRODUCTION
The paper addresses image caption generation by introducing a large human-caption dataset and an evaluation server designed to make algorithm comparisons more consistent.
- Image caption generation is a long-standing, challenging AI problem spanning computer vision, natural language processing, and machine learning.
- The Microsoft COCO Caption dataset collects human-generated captions for images from the Microsoft COCO dataset using Amazon’s Mechanical Turk.Upon completion, it is expected to contain over a million captions.
- The evaluation server addresses inconsistent comparisons caused by differing metrics and metric implementations.It evaluates captions produced by different approaches using multiple automatic metrics.
- The reported metrics include BLEU, METEOR, ROUGE, and CIDEr.
- The paper describes data collection, evaluation metrics, human performance, annotation instructions, server submission procedures, future directions, and known issues.
2 DATA COLLECTION
The dataset reuses MS COCO’s image splits and gathers multiple human captions through Amazon’s Mechanical Turk, including five-reference and forty-reference evaluation sets.
- The caption dataset uses the original MS COCO training, validation, and testing image splits.
- The images were selected to contain multiple objects in natural contexts, making them challenging for image captioning.
- MS COCO c5 provides five reference captions for every image across the training, validation, and testing datasets.
- MS COCO c40 provides 40 reference sentences for 5,000 randomly selected testing images to improve correlation with human judgment.
- Captions were generated by human subjects on Amazon’s Mechanical Turk using instructions to describe important scene content and avoid speculation or irrelevant details.The instructions also prohibit proper names, dialogue descriptions, and sentences beginning with “There is.”
- 1,026,459 captions were collected across 82,783 training, 40,504 validation, and 40,775 testing images.The testing set includes 179,189 captions for MS COCO c5 and 200,060 for MS COCO c40.
3 CAPTION EVALUATION
The evaluation server accepts candidate captions and applies consistent preprocessing and multiple automatic metrics across validation and testing data. It includes BLEU, ROUGE, METEOR, and CIDEr-D, each capturing different forms of overlap, alignment, or consensus, while BLEU is limited for sentence-level comparison.
- The server receives candidate captions for validation and testing, keeping testing references private to reduce overfitting.
- It evaluates captions with BLEU-1 through BLEU-4, ROUGE-L, METEOR, and CIDEr-D on both MS COCO c5 and c40.
- Candidate and reference captions are tokenized with Stanford PTBTokenizer and stripped of punctuation before scoring.
- 3.3 BLEU: BLEU measures clipped n-gram precision with a brevity penalty and combines n-gram precisions using a weighted geometric mean.
- 3.3 BLEU: BLEU performs well for corpus-level comparisons but poorly for individual sentences because higher-order n-gram matches are rare.
- ROUGE and METEOR: ROUGE uses n-gram recall, longest-common-subsequence matching, or skip bi-grams, while METEOR aligns tokens using exact matches, synonyms, stems, and paraphrases.
- 3.6 CIDEr: CIDEr weights n-grams with TF-IDF, averages candidate-reference cosine similarities, and CIDEr-D adds clipping and a length-based Gaussian penalty to resist gaming.
4 HUMAN PERFORMANCE
The paper measures human agreement in image-caption word prediction and models its precision-recall behavior using a simplified word-use model. Human agreement exhibits stable recall with precision depending on the number of reference captions, motivating Precision at Human Recall.
- 4.1 Interhuman Agreement: Human agreement is evaluated by treating one additional human caption as a prediction against the other reference captions.The evaluation server computes caption metrics for this held-out human caption, with results reported in Table 1.
- 4.2 Human Agreement for Word Prediction: Human precision and recall are computed from how many of k subjects use a word to describe images across the dataset.The calculation uses weighted positive and negative images based on the number of captions containing the word.
- 4.2 Human Agreement for Word Prediction: Human agreement points for nouns, verbs, adjectives, and all words lie approximately on a one-dimensional precision-recall curve.Words with high recall also have high precision; nouns such as “elephant” have high recall, while adjectives such as “bright” have low recall.
- 4.2 Human Agreement for Word Prediction: The model assumes word use depends on object presence through p and excludes word use when the corresponding object is absent.It ignores image-level saliency and assumes P(w = 1|o = 0) = 0.
- 4.2 Human Agreement for Word Prediction: Weighted precision and recall depend only on p, with recall exactly equal to the probability p that a subject uses the word when its object is present.The expressions are independent of q, the probability that the object occurs in an image.
- 4.2 Human Agreement for Word Prediction: With k = 4, the model closely matches empirical precision-recall tradeoffs; reducing reference captions preserves recall but lowers precision.The model also predicts that human precision approaches one with infinitely many subjects.
- 4.2 Human Agreement for Word Prediction: The fixed human recall motivates Precision at Human Recall, which compares system precision at human recall with human precision.This metric accounts for precision variation caused by the number of annotations.
5 EVALUATION SERVER INSTRUCTIONS
The evaluation server requires standardized caption submissions for validation and testing datasets, using specified JSON files and controlled training procedures. Results are uploaded for server evaluation, while validation scores can also be computed locally.
- Submission Procedure: Participants must generate results for both validation and testing datasets using the prescribed training and tuning restrictions.Test results use the permitted training data, while validation tuning is restricted to the training and validation datasets.
- Submission Format: Each dataset requires a JSON file containing image ids and candidate captions.The required record format is [{ “image id” : int, “caption” : str, }].
- Submission Procedure: Results are uploaded in a zip file, and validation evaluation is also available through provided GitHub code.The number of submissions per user is limited to a fixed amount.
6 DISCUSSION
The discussion highlights that human captions vary substantially and that automatic metrics may not align with human judgment. The authors therefore propose human evaluation to assess caption quality and metric validity.
- Challenges: Human subjects can produce substantially different captions that people may nevertheless judge equally good.This variability makes automatic evaluation of image captions difficult.
- Future Research: The authors expect validation results to support future research in image captioning.
- Evaluation Limitations: Because automatic metrics do not always correspond to human judgment, the authors plan human studies of caption quality, grammar, and metric correlation.These studies are intended to determine whether algorithmic improvements reflect genuine progress or overfitting to a specific metric.