Source-linked AI summary
SPICE: Semantic Propositional Image Caption Evaluation
Peter Anderson, Basura Fernando, Mark Johnson, Stephen Gould
TL;DR
Image-caption evaluation metrics based on n-gram overlap do not reliably capture semantic equivalence or human judgment. The paper introduces SPICE, which compares scene-graph representations of candidate and reference captions, and reports stronger agreement with human evaluations than existing metrics. SPICE also supports analyses of semantic capabilities such as color understanding and counting.
Problem
Existing automatic caption metrics are primarily sensitive to n-gram overlap, although n-gram overlap is neither necessary nor sufficient for conveying the same meaning.
Method
SPICE transforms candidate and reference captions into scene graphs encoding semantic propositional content, then compares their object, attribute, and relation tuples with an F-score.
Results
SPICE outperforms Bleu, METEOR, ROUGE-L, and CIDEr in agreement with human evaluations of model-generated captions.
Takeaways & Limitations
SPICE enables more detailed evaluation of caption generators, including analysis of color understanding and counting.
Takeaways & Limitations
SPICE neglects fluency and assumes captions are well-formed, while further improvements depend on advances in semantic parsing.
Abstract
from arXiv · showhide
There is considerable interest in the task of automatically generating image captions. However, evaluation is challenging. Existing automatic evaluation metrics are primarily sensitive to n-gram overlap, which is neither necessary nor sufficient for the task of simulating human judgment. We hypothesize that semantic propositional content is an important component of human caption evaluation, and propose a new automated caption evaluation metric defined over scene graphs coined SPICE. Extensive evaluations across a range of models and datasets indicate that SPICE captures human judgments over model-generated captions better than other automatic metrics (e.g., system-level correlation of 0.88 with human judgments on the MS COCO dataset, versus 0.43 for CIDEr and 0.53 for METEOR). Furthermore, SPICE can answer questions such as `which caption-generator best understands colors?' and `can caption-generators count?'
1 Introduction
The paper argues that n-gram overlap inadequately reflects caption meaning and introduces SPICE, which evaluates semantic propositional content through scene graphs. The metric represents objects, attributes, and relations, then compares candidate and reference tuples to better match human judgments.
- Motivation: Existing caption metrics rely primarily on n-gram overlap, which is neither necessary nor sufficient for equivalent meaning.
- Method: Scene graphs encode objects, attributes, and relations, and SPICE computes an F-score over matching tuples.
- Motivation: Captions with different meanings can receive high n-gram similarity, while semantically equivalent captions can receive low similarity.The paper contrasts shared wording in captions about a girl and a giraffe with paraphrased captions about a pot and pan.
- Method: SPICE evaluates semantic propositional content by transforming candidate and reference captions into scene graphs.Scene graphs abstract away most lexical and syntactic variation while encoding caption meaning.
- Results: SPICE outperforms Bleu, METEOR, ROUGE-L, and CIDEr in agreement with human evaluations across datasets and human-evaluation settings.Its performance can also be decomposed to examine properties such as color understanding and counting.
2 Background and Related Work
Prior caption-evaluation metrics mainly compare candidate and reference sentences through n-gram matching, while related work explores semantic role labels and scene-graph representations. These approaches motivate semantic evaluation beyond surface word overlap.
- Caption Evaluation Metrics: Caption metrics are generally similarity measures comparing candidate sentences with reference or ground-truth sentences.
- Caption Evaluation Metrics: Bleu, METEOR, ROUGE, and related metrics primarily use n-gram matching, with METEOR additionally incorporating stem, synonym, and paraphrase matches.
- Caption Evaluation Metrics: Human-correlation studies report varied performance for n-gram metrics across PASCAL 1K, Flickr 8K, and related datasets.Reported examples include Bleu-1 correlations of -0.17 and 0.05 on PASCAL 1K and METEOR correlation of 0.524 on Flickr 8K.
- Semantic Representations: Semantic role-labeling metrics represent basic event structure by matching semantic frames across sentences.
- Semantic Representations: Scene graphs have been used to improve complex image and video retrieval queries and can be parsed from natural-language descriptions.
- Semantic Representations: Abstract meaning representations and Smatch provide another semantic-graph framework, but initial experiments found AMR with Smatch performed poorly for image-caption evaluation.
3 SPICE Metric
SPICE evaluates captions by comparing scene-graph representations of candidate and reference captions. It matches semantic propositions about objects, attributes, and relations, then computes an interpretable F-score while assuming well-formed captions.
- SPICE disregards the image at evaluation time and compares candidate and reference captions through scene descriptions, giving primacy to nouns.
- Semantic Parsing: Scene graphs encode object mentions, object relations, and object attributes using open-world vocabularies that can include nouns such as grass and sky.
- Semantic Parsing: SPICE represents object counts as attributes rather than separate object instances, simplifying graphs but losing collective-versus-distributive distinctions.
- Semantic Parsing: The parser establishes dependency syntax, applies post-processing, and uses nine linguistic rules to extract scene-graph elements.
- Gameability: SPICE is not tied to its parsing pipeline, but it neglects fluency and may be gamed by captions containing semantic elements without grammatical quality.
- F-score Calculation: Semantic propositions become tuples, matching uses lemmatization or WordNet synonyms, and SPICE combines tuple precision and recall in an F-score.
4 Experiments
The experiments compare SPICE with existing metrics using system- and caption-level correlations against human judgments across multiple caption datasets and evaluation formats. On COCO, SPICE tracks overall and dimension-specific human judgments more accurately than the alternatives.
- The evaluation studies system-level and caption-level correlations between automatic metrics and human judgments across four datasets and varied captioning models.
- System-level correlations use Pearson’s coefficient, while caption-level correlations use Kendall correlation over dataset caption pairs.
- SPICE more accurately reflects overall quality and correctness, detailedness, and saliency; it is the only metric reported to reward caption detail.
- COCO human evaluations cover overall quality, Turing-test performance, correctness, detail, and similarity to human descriptions.
4.1 Datasets
The experiments draw on COCO and Flickr 8K resources with human-generated references and human quality judgments. COCO additionally supplies challenge-entry judgments and a fixed evaluation protocol.
- COCO contains 123,293 images, with 82,783 for training, 40,504 for validation, and 40,775 held out for testing.
- COCO images have five reference captions in C5, while 5,000 randomly selected test images have 40 captions in C40.
- COCO collected 255,000 AMT judgments across 15 competition entries plus human and random entries, covering overall quality, correctness, detail, and saliency.
- SPICE methodology was fixed before COCO evaluation, without access to COCO test captions.
- Flickr 8K contains 8,092 images with five human reference captions each and graded quality scores for 5,822 captions.
Flickr 8K.
The composite dataset aggregates 11,985 human judgments across Flickr 8K, Flickr 30K, and COCO captions. It uses graded correctness scores and includes human references alongside captions from two recent models.
- The composite dataset contains 11,985 human judgments over Flickr 8K, Flickr 30K, and COCO captions.
- Captions receive AMT correctness scores from 1, indicating no relevance, to 5, indicating a perfect relation to the image.
- Candidate captions come from human references and two recent captioning models.
PASCAL-50S
PASCAL-50S contains dense human annotations and judgments for comparing candidate captions, but its evaluation protocol differs from image-grounded caption assessment.
- PASCAL-50S comprises 1,000 images from 20 classes, each expanded from five captions to 50 human-authored captions.The captions were collected using AMT from images in the UIUC PASCAL Sentence Dataset.
- The dataset includes human judgments for 4,000 candidate sentence pairs, covering human-correct, human-incorrect, human-model, and model-model comparisons.Each pair was evaluated against 48 of the 50 possible reference captions.
- Workers judged which candidate was more similar to a reference sentence rather than evaluating captions against the image.The study notes that variation in reference-caption quality may introduce noise, though the difference from prior protocols was not studied.
4.2 System-Level Correlation
Across 2015 COCO Captioning Challenge entries, SPICE aligns with human quality judgments more strongly than competing metrics and uniquely identifies human captions as best.
- 0.88 is SPICE’s correlation with human quality judgments, exceeding CIDEr at 0.43 and METEOR at 0.53.The comparison uses system-level judgments over the challenge entries.
- SPICE more accurately reflects human judgment overall and across correctness, detailedness, and saliency dimensions.Only SPICE rewards caption detail; Bleu and ROUGE-L appear to penalize detailedness, while CIDEr and METEOR results are not statistically significant.
- SPICE is the only metric that ranks human-generated captions first among the 15 challenge entries.CIDEr ranks human captions seventh and METEOR ranks them fourth.
- SPICE is also the only metric that correctly selects the top five non-human entries.Figure 3 compares metric scores with human judgments for individual challenge models.
4.3 Color Perception, Counting and Other Questions
SPICE’s decomposable semantic tuples support targeted analysis of caption-generator abilities, revealing strong color performance but weak evidence of counting.
- SPICE tuples can be subdivided into objects, attributes, and relations, enabling precision, recall, and F-score analysis by semantic category.The tuples can also be subdivided more finely for targeted analyses.
- Table 2 isolates color, count, and size attribute tuples to assess specific caption-generator capabilities.The analysis uses word lists for colors, numbers one through ten, and size-related adjectives.
- The MSR entry exceeds the human F-score baseline for tuples containing color attributes.MSR incorporates specifically trained visual detectors for nouns, verbs, and adjectives.
- There is less evidence that the evaluated models have learned to count objects.The table caption states that none exhibits a convincing ability to count.
4.4 Caption-Level Correlation
At the caption level, SPICE only modestly exceeds existing metrics on Flickr 8K and the composite dataset, while its advantage becomes clearer when judgments are aggregated. On PASCAL-50S, overall performance is similar, but SPICE best distinguishes model-generated captions.
- SPICE more closely approximates human judgment when scores are aggregated over more captions.
- 0.39 Kendall’s τ: SPICE exceeds CIDEr (0.36) and METEOR (0.35) on the composite dataset.Inter-human agreement cannot be established because the dataset provides one score per image-caption pair.
- Caption-level SPICE performance represents only a modest improvement over existing metrics relative to the 0.73 correlation between human scores.
- On PASCAL-50S, SPICE, METEOR, and CIDEr show little difference overall, but SPICE performs best on pairs of model-generated captions.Distinguishing better-performing algorithms is identified as the primary motivation for the work.
5 Conclusion and Future Work
SPICE evaluates captions through semantic recovery of objects, attributes, and relations, and experiments show it captures human judgments over model-generated captions better than existing n-gram metrics. Its performance remains bounded by semantic-parsing challenges.
- SPICE measures how effectively image captions recover objects, attributes, and relations between them.
- Experiments on natural image-captioning datasets show SPICE captures human judgment over model-generated captions better than existing n-gram metrics.
- Semantic-parsing challenges remain, limiting the metric’s current scope for further improvement.The authors expect more powerful parsers to support future improvements.
- Future work proposes human-annotated upper bounds for how closely SPICE could approximate human judgments with perfect semantic parsing.