Source-linked AI summary
Re-evaluating Automatic Metrics for Image Captioning
Mert Kilickaya, Aykut Erdem, Nazli Ikizler-Cinbis, Erkut Erdem
TL;DR
Image-captioning evaluation needs reliable automatic metrics because human evaluation is costly and existing measures can diverge from subjective judgments. This paper compares established metrics through correlation, accuracy, and distraction experiments while investigating WMD, finding that WMD performs strongly across these evaluations. The study also identifies metric differences and scope boundaries, including SPICE’s dependence on parsing quality.
Problem
Automatic image-captioning metrics can inadequately mimic human judgments, while human evaluations are costly and not repeatable.
Method
The paper conducts an in-depth evaluation of existing metrics and investigates Word Mover’s Distance using correlation, accuracy, distraction, and significance analyses.
Results
SPICE, METEOR, and WMD perform best for correlation, CIDEr and WMD for accuracy, and CIDEr, METEOR, and WMD are less affected by distractors.
Takeaways & Limitations
WMD is effective for image-caption evaluation, combining high human-score correlations with lower sensitivity to synonym swapping and strong accuracy and distraction performance.
Takeaways & Limitations
SPICE performance depends substantially on parsing quality, as illustrated by a parsing failure in which roles are assigned incorrectly.
Abstract
from arXiv · showhide
The task of generating natural language descriptions from images has received a lot of attention in recent years. Consequently, it is becoming increasingly important to evaluate such image captioning approaches in an automatic manner. In this paper, we provide an in-depth evaluation of the existing image captioning metrics through a series of carefully designed experiments. Moreover, we explore the utilization of the recently proposed Word Mover's Distance (WMD) document metric for the purpose of image captioning. Our findings outline the differences and/or similarities between metrics and their relative robustness by means of extensive correlation, accuracy and distraction based evaluations. Our results also demonstrate that WMD provides strong advantages over other metrics.
1 Introduction
Image captioning generates natural descriptions from images, combining visual understanding with language generation. As datasets grow, automatic metrics have become important but require closer analysis because they can diverge from human judgments.
- Task context: Image captioning generates natural descriptions of images and requires both visual understanding and linguistic generation.The task can also be framed as machine translation from a visual source domain to a target language such as English.
- Task context: Encoder-decoder approaches have achieved considerable success on FLICKR8K, FLICKR30K, and MS COCO benchmarks.
- Evaluation motivation: Automatic metrics became increasingly important because human evaluations are costly to acquire and not repeatable.They score similarity or dissimilarity between generated captions and human-written reference descriptions.
- Existing metrics: BLEU, ROUGE, METEOR, and TER originated in machine translation or summarization, whereas CIDEr and SPICE were developed specifically for image caption evaluation.
- Evaluation motivation: Existing metrics can inadequately mimic human judgments, including cases where models surpass a human upper bound automatically but not under subjective evaluation.These findings motivate an in-depth analysis of current image-description metrics.
- Study scope: The study evaluates metrics through human-correlation analysis, significance testing, and tests of whether metrics distinguish caption pairs.The Williams significance test is used to examine differences, similarities, and complementarity between metric results.
2 Evaluation Metrics
The paper reviews established and image-captioning-specific metrics, then introduces WMD as a semantic document-distance measure. It highlights how metric design affects sensitivity to synonyms, word order, parsing errors, and redundant content.
- Existing metrics: BLEU, ROUGE, METEOR, CIDEr, and SPICE evaluate caption similarity using different linguistic or scene-graph representations.BLEU, ROUGE, METEOR, and CIDEr compare words or n-grams, whereas SPICE uses scene graphs; WMD uses word embeddings.
- Existing metrics: ROUGE-L measures longest common subsequences and favors long sentences because it relies heavily on recall.
- Existing metrics: METEOR combines unigram precision and recall with synonym and paraphrase matching, but synonym matching may miss broader semantic similarity.
- Existing metrics: CIDEr measures consensus through tf-idf-weighted 1-4-gram cosine similarity, but its purely linguistic design can overweight unimportant details.
- SPICE: SPICE parses captions into object, relation, and attribute tuples, then computes F1 agreement with reference tuples using WordNet synonym matching.Its score depends on parsing quality, and an example shows swimming parsed as an object and dog as an attribute.
- WMD: WMD compares normalized bag-of-words captions by moving word mass between embedding vectors at minimum cumulative cost.Word-pair costs are Euclidean distances between word2vec embeddings, and the document distance is solved as an Earth Mover’s Distance optimization.
- Metric drawbacks: Small caption changes expose metric-specific weaknesses: synonyms reduce all scores, reordered words notably reduce BLEU, ROUGE, and CIDEr, while WMD and SPICE resist word-order changes.The example attributes SPICE failures to parsing or synonym matching and CIDEr failures to unbalanced tf-idf weighting.
3 Evaluation and Discussion
The paper evaluates automatic captioning metrics against human judgments using correlation, significance, accuracy, and distraction-based experiments. Results show complementary strengths across metrics, with WMD performing strongly and a unified WMD+SPICE+METEOR metric outperforming individual metrics.
- Quality: The study compares metric–human correlations on FLICKR-8K and COMPOSITE, using Pearson, Spearman, and Kendall measures.FLICKR-8K uses expert scores, whereas COMPOSITE averages correlations with correctness and thoroughness judgments.
- Quality: SPICE gives the highest-quality comparisons on FLICKR-8K, while WMD and METEOR generally perform better on COMPOSITE.Score distributions further show that SPICE identifies irrelevant captions well but struggles to distinguish bad captions from relatively better ones.
- Quality: Pairwise correlations are generally high, but metrics cluster by design: n-gram metrics, scene-graph SPICE, and embedding-based WMD form complementary groups.BLEU and ROUGE correlate highly, while correlations between n-gram metrics and SPICE or WMD are lower.
- Quality: Williams significance tests find significant correlation differences for all metric pairs at p < 0.05, including pairs with close correlation scores.The test accounts for the dependence among correlations computed on the same dataset.
- Accuracy: WMD gives the best classification scores in three of four PASCAL-50S cases, especially for fine-grained MM and HC judgments.CIDEr outperforms other metrics on ABSTRACT-50S in both HC and HI cases.
- Robustness: WMD performs best for three of four distraction categories and second best for Replace-Scene, while SPICE performs worst overall.METEOR and CIDEr are also relatively robust to distractors.
- Discussion: Across experiments, SPICE, METEOR, and WMD lead correlation results; CIDEr and WMD lead accuracy; CIDEr, METEOR, and WMD are less affected by distractors.The findings indicate that existing metrics have different strengths and weaknesses arising from their design choices.
- Discussion: WMD+SPICE+METEOR achieves Spearman correlations of 0.66 on FLICKR-8K and 0.45 on COMPOSITE, improving over SPICE’s 0.64 and 0.42.The unified metric significantly outperforms individual metrics according to Williams test at p < 0.01.
4 Conclusion
The paper carefully evaluates automatic image-captioning metrics, proposes WMD for caption evaluation, and shows that metrics differ significantly while WMD performs strongly across evaluation tasks.
- The paper provides a careful evaluation of automatic image-captioning metrics and highlights their drawbacks.
- WMD uses word2vec embeddings to compute semantic similarity between sentences for image-caption evaluation.
- The metrics are empirically shown to differ significantly from one another.
- The findings motivate further research into better, potentially learning-based evaluation metrics.