PaperlayerView exact revision on arXiv

Source-linked AI summary

CIDEr: Consensus-based Image Description Evaluation

Ramakrishna Vedantam, C. Lawrence Zitnick, Devi Parikh

arXiv:1411.5726v2cs.CVcs.CLcs.IR

TL;DR

Evaluating automatically generated image descriptions remains challenging, especially because human preferences may not align with human-likeness. The paper introduces a consensus-based protocol, CIDEr, and two 50-reference-sentence datasets; CIDEr achieves 84% accuracy on both PASCAL-50S and ABSTRACT-50S, improving consensus measurement over existing metrics.

  • Problem

    Evaluating automatically generated image descriptions remains challenging because human preferences may differ from how human-like descriptions are judged.

  • Method

    The paper combines triplet-based human consensus judgments, the CIDEr similarity metric, and two datasets containing 50 human reference descriptions per image.

  • Results

    84% accuracy on both PASCAL-50S and ABSTRACT-50S; CIDEr consistently captures consensus better than existing metrics.

  • Takeaways & Limitations

    The protocol enables objective comparison of image-generation approaches based on human-likeness without arbitrary weighting of content, grammar, or saliency.

  • Takeaways & Limitations

    Human studies were conducted on Amazon Mechanical Turk with subjects restricted to the United States and additional worker-history qualifications.

Abstract

from arXiv · show

Automatically describing an image with a sentence is a long-standing challenge in computer vision and natural language processing. Due to recent progress in object detection, attribute classification, action recognition, etc., there is renewed interest in this area. However, evaluating the quality of descriptions has proven to be challenging. We propose a novel paradigm for evaluating image descriptions that uses human consensus. This paradigm consists of three main parts: a new triplet-based method of collecting human annotations to measure consensus, a new automated metric (CIDEr) that captures consensus, and two new datasets: PASCAL-50S and ABSTRACT-50S that contain 50 sentences describing each image. Our simple metric captures human judgment of consensus better than existing metrics across sentences generated by various sources. We also evaluate five state-of-the-art image description approaches using this new protocol and provide a benchmark for future comparisons. A version of CIDEr named CIDEr-D is available as a part of MS COCO evaluation server to enable systematic evaluation and benchmarking.

1. Introduction

The paper addresses the difficulty of evaluating automatically generated image descriptions by introducing consensus as a measure of human-likeness. It proposes a human protocol, the CIDEr metric, and two 50-description-per-image datasets.

  • Motivation: Recent advances have increased interest in image description generation, but evaluating novel automatically generated sentences remains challenging and is critical for measuring progress.The motivation follows progress in object recognition, attribute classification, action classification, and crowdsourcing.
  • Consensus-based evaluation: Human judgments reveal that what people prefer often differs from what is human-like, motivating evaluation against the majority consensus of image descriptions.The protocol measures similarity between a candidate sentence and human-provided ground-truth sentences.
  • CIDEr: CIDEr measures generated-sentence similarity against human ground-truth sets and captures grammaticality, saliency, importance, precision, and recall through sentence similarity.The metric shows high agreement with consensus as assessed by humans.
  • Datasets: Five descriptions per image are insufficient to measure majority human descriptions, so the authors collect PASCAL-50S and ABSTRACT-50S with 50 descriptions per image.The datasets are introduced specifically to support accurate consensus measurement.
  • Contributions: The work contributes a consensus-based protocol, a human annotation modality, an automated metric, two datasets, comparisons across five state-of-the-art approaches, and public CIDEr evaluation support.The code and datasets are made available on the authors’ webpages, and CIDEr is provided through the MS COCO caption evaluation server.

2. Related Work

Prior work addresses image-language relationships, image-description generation, and automated text evaluation, while existing datasets typically provide limited annotations. The paper extends this landscape with consensus-oriented evaluation and datasets containing 50 sentences per image.

  • Vision and Language: Vision-language studies model objects, visual-textual relationships, prepositions, attributes, and co-occurring visual phrases.
  • Image Description Generation: Image-description generation methods are broadly retrieval-based or generation-based, including global retrieval, phrase stitching, and convolutional-recurrent models.
  • Automated Evaluation: BLEU, METEOR, and ROUGE have been applied to automated evaluation, but their correlation with human judgment is weak to moderate and BLEU’s effectiveness has been questioned.
  • Datasets: Existing image-description datasets include UIUC Pascal with 5 descriptions for 1,000 images and SBU with one description per image for one million images.
  • Datasets: PASCAL-50S provides 50 sentences for 1,000 UIUC Pascal images, while ABSTRACT-50S provides 50 sentences for 500 Abstract Scenes images.
  • Datasets: The paper argues that more sentences per image are essential for reliable automatic evaluation.

3. Consensus Interface

The consensus interface measures whether a candidate sentence matches how most people describe an image through relative human judgments. Workers compare candidate pairs against reference sentences, and majority-based rankings provide ground truth for evaluating automated metrics.

  • Human comparison interface: Workers see a reference sentence A and two candidate sentences B and C, then choose which candidate is most similar to A.The protocol forms triplets using all reference sentences for an image.
  • Human comparison interface: The task uses relative comparisons without explicitly defining “similarity,” and some workers reported imagining the described scene.This design asks which candidate is more similar rather than requiring an absolute similarity judgment.
  • Consensus aggregation: Three judgments are collected per triplet, with the majority vote determining the triplet outcome.The triplet-level majority vote aggregates the individual human judgments.
  • Consensus aggregation: Candidate B wins a pairwise comparison when it is judged more similar by a majority of triplets, and candidate C is treated analogously.These pairwise relative rankings are compared with the ordering produced by automated metric scores.

4. CIDEr Metric

CIDEr evaluates how well a candidate description matches the consensus of reference descriptions by comparing their stemmed n-grams. It uses TF-IDF weighting and combines average cosine similarities across n-gram lengths to emphasize informative, semantically rich agreement.

  • CIDEr Metric: CIDEr first reduces candidate and reference words to their stem or root forms and represents each sentence by its ordered n-grams.For example, “fishes,” “fishing,” and “fished” are reduced to “fish.”
  • CIDEr Metric: TF-IDF weighting increases the influence of n-grams frequent in an image’s references while discounting n-grams common across the dataset.The inverse-document-frequency component provides word saliency by reducing the weight of popular, less-informative terms.
  • CIDEr Metric: CIDErn computes average cosine similarity between the candidate and reference n-gram vectors, accounting for both precision and recall.The vectors contain the weighted n-grams of length n, and cosine similarity compares the candidate with each reference sentence.
  • CIDEr Metric: The metric combines scores from n-grams of varying lengths to capture grammatical properties and richer semantics, using uniform weights wn = 1/N with N = 4.Longer n-grams provide higher-order information, while the empirically selected weighting scheme gives each n-gram order equal weight.

5. New Datasets

The paper introduces PASCAL-50S and ABSTRACT-50S as consensus-based testing datasets for evaluating image caption generation. Each image has many human reference descriptions collected with instructions emphasizing objective, representative scene content.

  • Dataset construction: PASCAL-50S and ABSTRACT-50S provide 50 reference sentences per image for 1,000 and 500 images, respectively.They are intended as testing datasets that enable consensus-based evaluation.
  • Annotation procedure: Subjects were asked to describe what was happening in each image while capturing its main aspects objectively.The collection interface showed an image and a text box.
  • Annotation procedure: Instructions discouraged dialogue and overly descriptive sentences, encouraged descriptions useful for recognizing images among similar images, and required acceptable grammar.Poor-grammar work was subject to rejection.

6. Experimental Setup

The experiments evaluate whether CIDEr captures human consensus better than existing metrics and compare automatic image-description methods by consensus matching. They use controlled candidate-sentence pairs from ABSTRACT-50S and PASCAL-50S, established metrics, and five machine-generation approaches.

  • Goals: The experiments assess CIDEr against existing metrics for human-consensus judgment and compare state-of-the-art description approaches by how well their outputs match consensus.The stated goals are metric evaluation and comparison of automatic image-description methods.
  • Candidate Sentences: On ABSTRACT-50S, 48 of 50 sentences per image serve as references, while the remaining 2 form 400 candidate pairs: 200 human-correct and 200 human-incorrect.Candidate pairs correspond to HC and HI conditions in the triplet annotation.
  • Candidate Sentences: On PASCAL-50S, 4,000 candidate pairs combine human sentences with outputs from five methods spanning retrieval- and generation-based approaches.The methods are Midge, Babytalk, Story, Video, and Video+.
  • Candidate Sentences: PASCAL-50S pairs comprise human–human correct, human–human incorrect, human–machine, and machine–machine comparisons, with machine outputs sampled so methods participate roughly equally.The four pair types compare agreement within and across human- and machine-generated descriptions.
  • Metrics: Evaluation compares BLEU1, BLEU4, ROUGE1, ROUGES, ROUGEL, and METEOR with CIDEr.BLEU is precision-based, ROUGE is recall-based, and ROUGES and ROUGEL use skip-bigrams and longest common subsequence, respectively.
  • Machine Approaches: Machine approaches are compared on 100 UIUC Pascal Sentence Dataset images having outputs from all five evaluated methods, using all 5C2 machine-method pairs per image.The five methods are Midge, Babytalk, Story, Video, and Video+.

7. Results

CIDEr most accurately matched human consensus across the PASCAL-50S and ABSTRACT-50S datasets, outperforming established metrics and approaching human performance. It also produced rankings of image-description systems that closely agreed with human judgments.

  • Metric comparison: At 48 reference sentences, CIDEr was the best-performing metric on both ABSTRACT-50S and PASCAL-50S.With only 5 sentences, CIDEr and METEOR still outperformed BLEU and ROUGE; CIDEr was better on ABSTRACT-50S, while METEOR was better on PASCAL-50S.
  • Metric comparison: 84% accuracy was achieved by CIDEr on both PASCAL-50S and ABSTRACT-50S, consistently capturing consensus across datasets.ROUGE1 and BLEU1 were less effective at capturing consensus.
  • Metric comparison: 84% accuracy with CIDEr at 48 sentences exceeded BLEU1’s 76% and ROUGE1’s 74% at 5 sentences, approaching human performance of 90%.The comparison uses PASCAL-50S accuracy values and reports the resulting proximity to human performance.
  • Candidate-pair evaluation: CIDEr was the best metric in five of six candidate-pair cases, with significant gains on the challenging machine–machine and human–human-correct tasks.The evaluated pair types included human–human correct, human–human incorrect, human–machine, and machine–machine pairs.
  • Description-system evaluation: 0.98 Pearson correlation measured agreement between human and CIDEr rankings of image-description approaches.Midge ranked highest and Story lowest by both humans and CIDEr, while the rankings of the two video approaches differed.

8. Gameability and Evaluation Server

The authors introduce CIDEr-D to reduce metric gaming while preserving close agreement with CIDEr, and make it available through the MS COCO caption evaluation server for systematic benchmarking.

  • Gameability: CIDEr-D modifies CIDEr to defend against gaming that can produce high-scoring but human-poor descriptions.The proposed changes remove stemming and penalize repeated high-confidence words or n-grams beyond their reference frequency.
  • Gameability: CIDEr-D uses a Gaussian penalty based on candidate–reference length differences, with σ = 6 and a factor of 10 for numerical comparability.The metric also penalizes repetition of specific n-grams beyond the number of times they occur in the reference sentence.
  • Gameability: 0.94 Spearman’s ρ rank correlation was observed between CIDEr-D and the original CIDEr, while CIDEr-D was more robust to gaming.Uniform weights are used, as in CIDEr.
  • Evaluation Server: CIDEr-D was made available in the MS COCO caption evaluation server to support systematic evaluation and benchmarking of image description approaches.The server evaluates approaches based on consensus.

9. Conclusion … Appendix V : Sentence Collection Interface

The paper concludes by introducing a consensus-based protocol, CIDEr, and two 50-sentence datasets for evaluating image descriptions. Its appendices compare annotation modalities and metrics, illustrate rankings and annotation differences, and document dataset collection procedures.

  • 9. Conclusion: The proposed protocol measures consensus through a new annotation modality, the CIDEr metric, and PASCAL-50S and ABSTRACT-50S datasets containing 50 sentences per image.It enables objective comparison of machine-generated descriptions based on human-likeness without arbitrarily weighting content, grammar, or saliency.
  • Appendix Overview: The appendices cover metric comparisons, sentence rankings, CIDEr versus CIDEr-D, differences between pairwise and triplet judgments, metric formulations, and qualitative outputs.They also include benchmarks of metric versions on consensus and qualitative examples from evaluated image-description methods.
  • Appendix II : Ranking of Sentences: CIDEr’s top-ranked reference sentences show high consensus, while ROUGE favors more detailed sentences and BLEU’s top-ranked sentences are less consistent.Each reference is scored against the remaining 47 sentences for the image.
  • Appendix III : Difference between Human-like and What Humans Like: Pairwise judgments of what is “better” can differ from the sentences people write when describing images, distinguishing “human-like” from “what humans like.”The appendix presents qualitative examples of this distinction using consensus-based selections and pairwise preferences.
  • Appendix IV : Ranking of sentences - CIDEr and CIDEr-D: Spearman’s ρ=0.94 measures the high correlation between CIDEr and CIDEr-D sentence rankings, which are also qualitatively very similar.CIDEr-D additionally avoids the gaming effects described in Section 8.
  • Appendix V : Sentence Collection Interface: The ABSTRACT-50S and PASCAL-50S sentence-collection interface is shown alongside stringent rejection criteria illustrated for AMT subjects.Figures 8 and 9 document the interface and rejection examples used during dataset collection.

Appendix VI : Image Description Method Outputs

Appendix VI presents sample descriptions generated by the five image-description methods compared in the paper. On the shown image, Midge and Babytalk produce the better descriptions, consistent with the paper’s findings.

  • Compared methods: Five methods are compared: Midge, Babytalk, Story, Video, and Video+.Video and Video+ are two versions of Translating Video Content to Natural Language Descriptions.
  • Sample outputs: The appendix shows one sample image alongside descriptions generated by all five methods.The outputs are presented in Fig. 10.
  • Qualitative comparison: Midge and Babytalk produce the better descriptions for the shown image, consistent with the paper’s findings.This comparison is qualitative and specific to the displayed image.

Appendix VII : Other Metrics … METEOR

The appendix reviews BLEU, ROUGE, and METEOR as alternative metrics for comparing candidate image descriptions with reference consensus. These metrics use n-gram, sequence-structure, skip-bigram, or word-alignment mechanisms, while BLEU is limited for sentence-level comparisons.

  • Appendix VII : Other Metrics: Descriptions are represented as one- to four-word n-grams, with each word reduced to its stem or root form.Thus, “fishes,” “fishing,” and “fished” are normalized to “fish.”
  • BLEU: BLEU computes sentence-level n-gram co-occurrence scores against reference sentences using clipped precision and a brevity penalty.With multiple references, the closest reference length is used for the brevity penalty.
  • BLEU: BLEU combines individual n-gram precisions with a weighted geometric mean over n = 1, 2, 3, 4.The weights wn are typically held constant across n.
  • BLEU: At sentence level, BLEU performs poorly because higher-order n-gram matches rarely occur between individual sentences.It performs better for corpus-level comparisons where many n-gram matches exist.
  • ROUGE: ROUGE-N measures n-gram recall, while ROUGE-L uses longest common subsequence structure and an F-measure combining precision and recall.ROUGE-L’s LCS allows words between shared words, unlike contiguous n-grams.
  • ROUGE: ROUGE-S uses skip bi-grams and computes an F-measure from precision and recall, allowing ordered word pairs to span intervening words.In practice, component words occur at a distance of at most 4.
  • METEOR: METEOR aligns candidate and reference words for 1:1 correspondence, minimizing contiguous ordered chunks through exact matches, WordNet synonyms, and stems.Its final score combines a precision-recall harmonic mean with a chunkiness penalty.

Appendix VIII : Detailed Evaluation

The appendix compares metric variants, including BLEU, ROUGE, and CIDEr, and examines how their rankings align with human consensus. It also contrasts CIDEr with CIDEr-D and reports metric-version performance on PASCAL-50S and ABSTRACT-50S.

  • Metric Variants: CIDEr exp combines scores across varying n-gram counts exponentially, whereas CIDEr max takes the maximum across scores with different reference sentences.These are evaluated alongside versions of BLEU and ROUGE; only the latest METEOR version is included.
  • Human Alignment: CIDEr better captures human consensus than ROUGE and BLEU, which respectively favor longer detailed sentences and shorter sentences lacking coverage without Brevity Penalty.The comparison uses ROUGE1 and BLEU1 versions.
  • CIDEr-D: CIDEr1 and CIDEr-D1 produce mostly similar rankings, while CIDEr-D is more robust to gaming effects than CIDEr.The comparison ranks 48 sentences from highest to lowest score.
Loading 1411.5726v2…