PaperlayerView exact revision on arXiv

Source-linked AI summary

CLIPScore: A Reference-free Evaluation Metric for Image Captioning

Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, Yejin Choi

arXiv:2104.08718v3cs.CVcs.CL

TL;DR

Reference-based caption metrics can diverge from human judgments, motivating evaluation without human-written references. The paper introduces CLIPScore and finds high human correlation for literal image-description tasks, with RefCLIPScore improving correlation further when references are available.

  • Problem

    Reference-based caption scoring can leave a gap from human quality judgments, motivating evaluation based directly on images and candidate captions.

  • Method

    The paper applies pretrained CLIP directly to image–generated-caption pairs and extends it with RefCLIPScore when reference captions are available.

  • Results

    For literal image-description tasks, CLIPScore achieves high correlation with human judgments without references, while RefCLIPScore improves correlation further.

  • Takeaways & Limitations

    CLIP can serve as a strong reference-free automatic metric for literal image-description tasks and a reasonable option for alt-text evaluation.

  • Takeaways & Limitations

    For news image captioning, the authors do not advocate reference-free evaluation because reference-based n-gram metrics correlate highly with human judgments.

Abstract

from arXiv · show

Image captioning has conventionally relied on reference-based automatic evaluations, where machine captions are compared against captions written by humans. This is in contrast to the reference-free manner in which humans assess caption quality. In this paper, we report the surprising empirical finding that CLIP (Radford et al., 2021), a cross-modal model pretrained on 400M image+caption pairs from the web, can be used for robust automatic evaluation of image captioning without the need for references. Experiments spanning several corpora demonstrate that our new reference-free metric, CLIPScore, achieves the highest correlation with human judgements, outperforming existing reference-based metrics like CIDEr and SPICE. Information gain experiments demonstrate that CLIPScore, with its tight focus on image-text compatibility, is complementary to existing reference-based metrics that emphasize text-text similarities. Thus, we also present a reference-augmented version, RefCLIPScore, which achieves even higher correlation. Beyond literal description tasks, several case studies reveal domains where CLIPScore performs well (clip-art images, alt-text rating), but also where it is relatively weaker in comparison to reference-based metrics, e.g., news captions that require richer contextual knowledge.

1 Introduction

Reference-based image-caption metrics can diverge from human judgments despite multiple human captions, motivating reference-free evaluation based only on the image and candidate text. The paper shows that CLIP supports this approach, introduces RefCLIPScore, and identifies domains where CLIPScore is strong or weaker.

  • Motivation: Reference-based scoring can remain insufficient even with multiple human-authored captions, leaving a gap from human quality judgments.The introduction attributes this limitation partly to the expense of collecting references and the shortcomings of reference-based comparison.
  • Reference-free evaluation: CLIPScore applies pretrained vision-language relationships directly to image-generated-caption pairs, achieving surprisingly high correlation with human judgments on standard benchmarks.The paper hypothesizes that models such as CLIP can support reference-free evaluation and reports this result on benchmarks including MSCOCO.
  • Contributions: RefCLIPScore is a reference-augmented version that achieves even higher human correlation than CLIPScore.The paper also tests CLIP-S on adversarial captions and never-before-seen images.
  • Robustness: CLIP-S is sensitive to adversarial noun-phrase swaps and reconstructs human judgments for images never posted publicly online.These experiments test robustness to plausible but incorrect distractors and generalization beyond publicly available imagery.
  • Case studies: CLIP-S correlates highly with Twitter alt-text quality ratings and reasons about clipart images, but reference-based methods correlate best for news captions.For emotive captions inspired by social-media language, even reference-based metrics fall short.

2 Related Work

Prior image-caption evaluation has primarily used reference-based metrics, while newer approaches incorporate image-text grounding, self-retrieval signals, or reference-free evaluation. These developments motivate evaluating captions through direct image-text compatibility rather than relying only on human-written references.

  • Reference-only image caption evaluation: Standard image-caption evaluation uses five reference-based metrics: BLEU-4, ROUGE-L, METEOR, CIDEr, and SPICE.They measure precision, recall, word-level alignment, n-gram tf-idf similarity with stemming, and scene-graph similarity, respectively.
  • Reference-only image caption evaluation: BERTScore was specifically tuned to image caption generation, yielding the domain-specific BERT-S++ method.The paper refers to the tuned method as BERT-S++.
  • Reference+image caption evaluation: Reference-plus-image metrics add image-text grounding to reference comparisons: TIGEr uses SCAN, whereas ViLBERTScore-F uses ViLBERT fine-tuned on 12 downstream vision-and-language tasks.These methods extend reference-based evaluation with pretrained multimodal representations.
  • Self-retrieval for image captioning: Self-retrieval approaches use the intuition that good captions should uniquely identify their images, while CLIP-S instead serves as an extrinsic evaluation metric.Self-retrieval loss can also indicate how distinctive captions are according to the generation model itself.

3 CLIPScore

Section 3 defines CLIPScore by using CLIP image–caption compatibility to evaluate candidate captions without references, with corpus scores obtained by averaging image–candidate similarities. When references are available, RefCLIPScore extends this approach by combining CLIP-S with reference similarity.

  • Model Details: CLIP is a cross-modal retrieval model trained on 400M image–caption pairs gathered from the web.The training data were collected using 500K search queries, with up to 20K image–caption pairs per query.
  • Evaluating Caption Generations with CLIP: CLIPScore evaluates a candidate by extracting image and caption features and computing their cosine similarity.Prefixing candidates with “A photo depicts” slightly improved correlations, while “A photo of” also worked well.
  • Evaluating Caption Generations with CLIP: Corpus-level CLIP-S averages image–candidate scores and does not depend on underlying references.With the ViT-B/32 backbone, roughly 4K image–candidate pairings can be processed per minute on a single consumer GPU and hard drive.
  • RefCLIPScore: RefCLIPScore incorporates available references by combining CLIP-S with the maximal reference cosine similarity through a harmonic mean.Reference representations are extracted using CLIP’s text transformer.

4 Benchmark Captioning Evaluations

Across literal image-description benchmarks, CLIP-S without references correlates strongly with human judgments, while RefCLIP-S generally improves correlation further. The results also show that visual and reference-based metrics capture complementary aspects of caption quality.

  • Caption-level evaluations: On caption-level literal photographic description tasks, CLIP-S without references achieves higher human-judgment correlation than previously proposed reference-based metrics, and RefCLIP-S improves it further.This pattern holds across Flickr8K-Expert, Flickr8K-CF, and Composite.
  • Pairwise evaluations: CLIP-S and RefCLIP-S generally achieve high performance across all four Pascal-50S caption-pair categories.Pascal-50S evaluates whether a metric selects the caption preferred by a majority of human annotators.
  • System-level evaluation: 0.59/0.63 and 0.69/0.74 are the system-level Spearman ρM1/ρM2 correlations for CLIP-S and RefCLIP-S, respectively, with all p < .05.The evaluation uses outputs from systems submitted to the 2015 MSCOCO Image Captioning Challenge, though the corpus contains only 12 datapoints.
  • Reference sensitivity: Over ten points is the increase in SPICE and BLEU-4 accuracy when shifting from one to four references, while CLIP-S outperforms all metrics except BERT-S (RoBERTa-F) with few references.RefCLIP-S works best in all cases, and CLIP-S and RefCLIP-S improve performance in the pairwise setting.
  • Human preference recovery: 86% of cases are correctly resolved by CLIP-S relative to a 50% random baseline and a 72% shorter-caption length baseline, versus 93% human agreement.The result comes from authors selecting which of two captions described image content more accurately.
  • Information gain: CLIP-S and reference-only metrics are complementary: CLIP-S or RefCLIP-S is selected first for information gain, while SPICE remains useful despite being image-unaware.The findings support reporting at least one image-aware metric such as CLIP-S alongside a strong reference-only metric such as SPICE.

5 Case Studies Using CLIPScore

Across four divergent image-description case studies, CLIP-S performs strongly for alt-text and offers meaningful visual-textual reasoning for clip-art, but struggles with non-literal engagingness and context-heavy news captions. Its strongest results occur when image-text compatibility is visually verifiable, while reference-based metrics remain preferable for richer contextual interpretation.

  • Alt-text quality: 48.4 τc correlation shows CLIP-S closely matches human alt-text quality judgments, outperforming BERT-S at 15 τc and the length baseline at 25 τc.The corpus contains 2.8K human judgments rating alt-text utility from 0 to 3.
  • Clip-art images: On non-photographic clip-art images, CLIP-S underperforms reference-based metrics but beats both baselines by a wide margin.The result indicates that CLIP-S captures meaningful visual-textual interactions beyond ordinary photographic scenes.
  • Engagingness: 65% of human judgments favored personality captions, while CLIP-S preferred traditional captions 80% of the time, revealing its bias toward literal descriptions.This study covered 2.4K pairwise ratings comparing traditional and personality-based captions.
  • Engagingness: 57% accuracy left CLIP-S no better than the 57% length-only baseline when comparing two generators, while BLEU-4, CIDEr, and SPICE reached 52%/53%/51%.Neither CLIP-S nor reference-based metrics reliably predicted human judgments of engagingness in this corpus.
  • News captions: 93 and 91 accuracies for METEOR and BLEU-4 surpassed CLIP-S at 65, which was only slightly above random on contextualized New York Times captions.CLIP-S succeeds for visually verifiable content but commonly fails when captions require information unavailable from the image alone.
  • News captions: The authors do not advocate reference-free evaluation for news captions because reference-based n-gram metrics achieve high correlation with human judgment.News captions frequently contain named entities and contextual interpretations rather than descriptions of visible scene objects.

6 Conclusion

CLIPScore performs well for reference-free evaluation of literal image descriptions, clip-art, and alt-text, but model-based evaluation inherits pretraining biases and raises social-bias concerns for deployment. Concurrent metrics provide related alternatives, with UMIC achieving similar literal-task correlations through synthetic negative-caption fine-tuning.

  • Conclusion: CLIPScore achieves high correlation with human judgments for literal image description without references and is a reasonable reference-free option for alt-text evaluation.Additional experiments suggest CLIP can also reason about non-photographic clip-art.
  • Limitations: Model-based reference-free metrics such as CLIP-S reflect biases in their pretraining data.The paper considers offline use for literal caption quality consistent with CLIP’s model-card recommendations.
  • Limitations: CLIP can make disproportionate incorrect classifications of people, including misclassifying male images into crime-related classes.The cited study motivates caution about social biases in candidate generations, especially for deployed systems.
  • Contemporaneous work: UMIC produces similar correlations with human judgment on literal image-description tasks compared to CLIP-S, using UNITER and synthetic negative-caption fine-tuning.FAIEr is another contemporaneous reference-free metric based on a pretrained object detector fine-tuned on MSCOCO.

A Evaluation and Replication Details

This section details replication efforts for established image-captioning evaluation protocols, emphasizing that preprocessing and reporting choices can substantially affect correlations with human judgments.

  • Replication motivation: Established corpora, metrics, and experimental settings became the standard for evaluating new image-captioning metrics, motivating a careful replication.The protocols were introduced by Anderson et al. (2016), but their standardization may have been inadvertent.
  • Replication motivation: Seemingly minor preprocessing and reporting choices can significantly change correlations with human judgments on these corpora.The replication effort explicitly examines how these choices affect evaluation results.

Flickr8K details

The Flickr8K evaluation settings contain discrepancies in rank-correlation handling and human-rating aggregation, which can substantially change reported results. The authors therefore standardize their results using τc correlation and aggregation method A, causing slight differences from prior work.

  • Evaluation discrepancies: Prior work mixed Kendall-C and Kendall-B, which handle frequent ties in discretized human Likert judgments differently.Three human ratings were collected for 5664 image–candidate pairs, and prior work generally flattened the 5664 × 3 judgments into one list for rank correlation.
  • Evaluation discrepancies: 44.9 is the replicated SPICE correlation under the original evaluation setting, while 52.9 results when using τc and aggregation method B.The increase reflects inflation caused by changing both correlation type and human-rating aggregation, and the same issue affects other metrics.
  • Evaluation protocol: The authors report results using τc correlation and human-rating aggregation method A, the most common setting.Consequently, their reported values may differ slightly from those in prior work.

Composite details

Composite-corpus results are sensitive to both Kendall correlation variant and whether ground-truth references are removed. Replication efforts suggest the original evaluations used τc with ground-truth references removed, while a BLEU-4 discrepancy may reflect smoothing differences.

  • Switching CIDEr from τb to τc increases rank correlation from 35 to 38 in this setting.
  • Composite-corpus comparisons are complicated by ambiguity over whether human-rated candidate captions sampled from references were removed.Each image originally had roughly five references, and prior work may have followed different curation standards.
  • Replication efforts suggest the original Composite evaluations used τc with ground-truth references removed.The authors reconstructed results under both reference-removal policies and identified this configuration as the likely original standard.
  • A discrepancy in BLEU-4 likely results from smoothing applied differently to individual captions versus the whole corpus.The paper attributes this possible issue to a smoothing difference noted in prior work.

MSCOCO system-level details

The MSCOCO system-level evaluation correlates caption metrics with human judgments using submissions from captioning systems. However, reliance on dev-set predictions and very small, potentially duplicated system samples undermines its value for fine-grained metric comparisons.

  • Evaluation setup: MSCOCO 2015 is a standard corpus for measuring system-level correlations between evaluation metrics and human judgments.The evaluation uses the MSCOCO test set.
  • Evaluation setup: Recent studies use dev-set predictions because test-set submissions are unavailable, assuming dev-set results correlate with test-set results.Earlier evaluation used a random sample of 1K test-set submissions from 15 teams, whereas later work used predictions from 12 teams.
  • Methodological concerns: Dev and test predictions may differ across teams, as illustrated by identical dev predictions but different test predictions for m-RNN submissions.The differing test predictions achieved significantly different scores.
  • Methodological concerns: Correlations over only 12, or possibly 11 duplicated, systems provide a weak basis for fine-grained comparisons between caption-generation metrics.The passage notes that Spearman and Pearson correlations over so few observations are simple to compute but problematic for this purpose.

Pascal-50S Setup Erratum

A replication effort uncovered that the original work used a nonstandard Pascal50S judgment set. The main-paper results were updated to use the intended 48 judgments, while 11-judgment results are provided separately for reproducibility.

  • Erratum: The original work used a different set of human judgments than the usual Pascal50S setup.This discrepancy was reported in March 2022 during replication and identified upon further investigation.
  • 11-judgment results: Table 7 reports Pascal50S-11-judgment accuracy results using 5 references and the non-standard 11-human-judgment version.It includes HC, HI, HM, and MM categories and averages results over 5 random samples; CLIP-S is unchanged because it does not use references.
  • Judgment sets: Pascal50S provides 11 human judgments per pair in pair_pascal.mat and 48 in consensus_pascal.mat, with the 48 judgments intended for evaluation.The main-paper results were updated accordingly.

B Rescaling CLIPScore

The authors rescale CLIP ViT-B/32 cosine similarities for readability and metric comparability without changing rankings. They multiply similarities by 2.5, clamp negatives to zero for RefCLIPScore, and caution that the parameters are model-specific.

  • Rescaling rationale: A monotonic rescaling of raw CLIP ViT-B/32 cosine similarities improves readability and aligns values more closely with other evaluation metrics without affecting rankings.Figure 4 shows raw candidate-reference and candidate-visual-content similarity distributions.
  • RefCLIPScore safeguard: Negative cosine similarities were not observed, but values are clamped to zero because RefCLIPScore’s harmonic mean is undefined for negatives.This safeguard is applied before computing RefCLIPScore.
  • Rescaling rationale: Multiplying by 2.5 stretches the CLIPScore distribution toward zero to one, although CLIPScore can exceed 1.The same weighting is retained for RefCLIPScore to balance image-candidate and reference-candidate similarity factors.
  • Model dependence: The exact rescaling parameters apply only to CLIP ViT-B/32 because future larger CLIP models may have different cosine-similarity distributions.The paper gives the unreleased ViT-L/14 variant as an example of a potentially different model.
Loading 2104.08718v3…