Source-linked AI summary
Learning to Evaluate Image Captioning
Yin Cui, Guandao Yang, Andreas Veit, Xun Huang, Serge Belongie
TL;DR
Image-captioning metrics often correlate imperfectly with human judgments and have blind spots for pathological captions. The paper trains a discriminator to distinguish human from machine-generated captions and augments training with pathological negatives. Experiments report greater robustness and stronger human correlation than existing metrics across Flickr 8k and COCO, supporting its complementary use with rule-based metrics.
Problem
Common image-captioning metrics often correlate poorly with human judgments and cannot flexibly address their known blind spots for pathological caption constructions.
Method
The paper learns a binary discriminator over image and caption representations and adds generated pathological sentences as negative training examples.
Results
The proposed metric is reported to improve robustness to pathological transformations and human correlation on Flickr 8k and COCO relative to commonly used metrics.
Takeaways & Limitations
The learned metric is presented as complementary to rule-based metrics, particularly for pathological cases that are easy to generate but difficult to capture with rules.
Takeaways & Limitations
The study does not account for different annotator personalities and identifies gameability as a concern for learning-based and rule-based metrics.
Abstract
from arXiv · showhide
Evaluation metrics for image captioning face two challenges. Firstly, commonly used metrics such as CIDEr, METEOR, ROUGE and BLEU often do not correlate well with human judgments. Secondly, each metric has well known blind spots to pathological caption constructions, and rule-based metrics lack provisions to repair such blind spots once identified. For example, the newly proposed SPICE correlates well with human judgments, but fails to capture the syntactic structure of a sentence. To address these two challenges, we propose a novel learning based discriminative evaluation metric that is directly trained to distinguish between human and machine-generated captions. In addition, we further propose a data augmentation scheme to explicitly incorporate pathological transformations as negative examples during training. The proposed metric is evaluated with three kinds of robustness tests and its correlation with human judgments. Extensive experiments show that the proposed data augmentation scheme not only makes our metric more robust toward several pathological transformations, but also improves its correlation with human judgments. Our metric outperforms other metrics on both caption level human correlation in Flickr 8k and system level human correlation in COCO. The proposed approach could be served as a learning based evaluation metric that is complementary to existing rule-based metrics.
1. Introduction
Image-captioning metrics often correlate poorly with human judgments and miss pathological sentence constructions. The paper proposes a learned discriminator, strengthened with targeted pathological examples, to evaluate captions in context.
- Motivation: Common metrics rely mainly on word overlap, while SPICE compares scene graphs and can better correlate with human judgments.These approaches nevertheless leave semantic or structural aspects insufficiently captured.
- Motivation: Existing metrics can correlate poorly with human judgments and exhibit blind spots that rule-based methods cannot easily repair.SPICE, for example, captures semantic meaning but tends to ignore syntactic quality and may favor repetitive long sentences.
- Approach: The proposed metric trains a binary classifier to distinguish human-written from machine-generated captions using image and caption representations.Its architecture combines a CNN for image features, an RNN with LSTM cells for captions, and a binary classifier as the critique.
- Approach: Pathological sentences are generated as negative training examples so the metric can adapt to targeted failure cases.The augmentation scheme creates varied unnatural constructions that may otherwise receive high evaluation scores.
- Results: The paper reports better correlation with human judgments on COCO and Flickr 8k than commonly used metrics, alongside robustness evaluations.The contributions also include studies of training factors and robustness to pathological transformations.
2. Related Work
Prior captioning metrics are generally based on similarity between candidate and reference captions, leaving semantic information and image context insufficiently represented. The proposed approach differs by incorporating image features and focusing its discriminator on evaluation rather than caption generation.
- Captioning evaluation: BLEU, METEOR, ROUGE, and CIDEr mainly use n-gram overlap and tend to be insensitive to semantic information.Human evaluation is reliable but costly, motivating the use of automatic metrics.
- Captioning evaluation: SPICE uses scene-graph similarity and correlates better with human judgments, but encounters difficulties with repetitive sentences.These metrics rely on candidate-reference similarity without incorporating the image itself.
- Adversarial training and evaluation: Existing adversarial evaluation approaches differ from this work because their discriminators assess generator performance, whereas this discriminator evaluates captions.The paper uses adversarial-style discrimination for metric learning rather than for generation.
3. Discriminative Evaluation
The learned critique scores captions by distinguishing human-written from machine-generated text in image context, using encoded image, reference-caption, and candidate-caption features. Data augmentation adds pathological captions as negative examples so the metric can target weaknesses such as word-frequency shortcuts, mismatched image content, and disrupted sentence structure.
- Evaluation Metric: The metric trains a critique to distinguish human-written captions from machine-generated captions, then scores candidates by their success in fooling it.The classifier outputs human-written or machine-generated probabilities for candidate captions.
- Model Architecture: The model encodes the image with a residual network and captions with LSTMs, combines context and candidate features, and classifies the result.The reference and candidate captions use LSTM encoders with shared weights; feature combination can use concatenation with an MLP or Compact Bilinear Pooling.
- Evaluation Metric: The score is conditioned on the reference image to assess whether a candidate captures image content and emphasizes important aspects.A reference caption can also be included as additional context.
- Data sampling and augmentation: Data augmentation generates pathological image-caption pairs as negative examples through random captions, word permutations, and random word replacements.These transformations respectively target image relevance, sentence structure, and rare-word behavior.
- Data sampling and augmentation: Random word replacement addresses a shortcut where a discriminator may treat rare words as evidence that a caption is human-written.Machine and human captions have markedly different word-frequency distributions, and unaugmented critiques may misclassify random words.
- Data sampling and augmentation: Transformation strength γ ranges from the original caption at γ% = 0 to the strongest transformation at γ% = 1.Robustness is evaluated on transformed captions across these transformation strengths.
4. Experiments
Experiments evaluate the learned metric on COCO and Flickr 8k for discrimination, robustness to pathological transformations, and correlation with human judgments. The metric benefits from image context, nonlinear feature interactions, and data augmentation, achieving strong robustness and human correlation.
- Experimental setup: The experiments use COCO for validation, testing, parameter tuning, and system-level evaluation, with roughly five human annotations per image.The COCO split contains 113,287 training images and validation and test sets of 5,000 images each.
- 4.2. Capability: All model variants assign much higher scores to human captions than machine-generated captions, demonstrating discrimination capability.The variants use generated captions from Show and Tell, Show, Attend and Tell, and NeuralTalk, together with human captions.
- 4.2. Capability: Including image features clearly improves discrimination, while adding a reference caption provides no significant improvement over image features alone.The results suggest that the image supplies sufficient context for distinguishing human-written from machine-generated captions.
- 4.3. Robustness: Data augmentation exposes robustness failures in unaugmented metrics, while nonlinear interactions such as compact bilinear pooling or nonlinear classifiers address linear models’ limitations.Without augmentation, models give high scores to captions from other images, random words, and permuted words; concatenation with a linear classifier remains weak even with augmentation.
- 4.3. Robustness: The proposed metric outperforms other metrics on random and permuted words and remains reasonably robust to captions from similar images.CIDEr and SPICE perform well on captions from similar images but fall behind on sentence structure.
- 4.4. Caption Level Human Correlation: The learned metrics achieve the best caption-level correlation with human judgments in both Flickr 8k Expert and Crowd Flower annotations.This result holds despite training with a simpler binary objective and without human annotations, because negative examples are generated automatically.
- 4.5. System Level Human Correlation: The learned metric surpasses all other metrics by large margins in system-level human correlation, especially when trained with data augmentation.The metric aligns particularly well with human judgments for top-performing methods.
5. Conclusion and Future Work
The paper presents a learning-based image-captioning metric designed to distinguish human from machine-generated captions and adapt to targeted pathological cases. It reports robustness and improved human correlation, while identifying annotator heterogeneity and adversarial gameability as future concerns.
- The proposed metric distinguishes human-written from machine-generated captions while adapting to targeted pathological cases.
- Data sampling and augmentation train the metric to behave robustly against captions produced by pathological transformations.
- Extensive experiments show better robustness and correlation with human judgments than previous metrics.
- The metric is positioned as complementary to rule-based metrics when pathological cases are easy to generate but difficult for hand-crafted metrics to capture.
- The study does not model different human annotator personalities, and future work includes heterogeneous captions and adversarial robustness.
A. Implementation Details
The implementation extracts image features using a pretrained ResNet-152 and represents caption words with a vocabulary and learned embeddings.
- Image features come from a pretrained 152-layer ResNet applied to the original image without resizing or cropping.
- The vocabulary contains the 10,000 most frequent training-caption words appearing at least five times, plus an unknown-word token.
- Each vocabulary word is represented by a one-hot vector and encoded through a learned embedding matrix.
A.3. Training
Training balances positive and negative caption examples, evaluates candidates across reference captions, and examines model capacity and data augmentation. The experiments indicate that augmentation affects the task learned, while larger LSTM configurations offer no obvious capability benefit.
- Training: Positive examples pair different reference captions for the same image, while negative examples include generated and pathological captions.
- Evaluation: Candidate scores are averaged across evaluations using every reference caption for the image as context.
- Evaluation: Caption-generator evaluation trains for 10 epochs, uses pathological transformations and Monte Carlo sampling, and averages held-out candidate scores.
- Evaluation: Caption-level human correlation is computed with Kendall’s τ over paired human annotations and metric scores.
- Model capacity: Higher-capacity LSTM models show no obvious capability benefit, motivating one layer with 512-dimensional hidden features.
- Data augmentation: Models without augmentation can discriminate human from machine captions, but they learn a simpler task than augmented models.
- Model capacity: Figure 8 compares models by LSTM layer count and hidden feature size, with human and machine captions shown as evaluation categories.
C. Caption Evaluation Examples
Examples show that using the image as context helps the metric detect captions referring to wrong objects and reward captions semantically relevant to the image, while misleading visual information remains a failure source.
- Image context helps the metric recognize captions referring to wrong objects and score image-relevant captions highly.
- Misleading visual information accounts for typical failure cases of the metric.
D. System Level Human Correlation on COCO
The proposed metric shows stronger human correlation than competing metrics across most evaluated caption-quality dimensions. Its advantage is especially notable for caption detailness, which other metrics fail to capture well.
- The comparison includes human and generated-caption scores across multiple feature-combination and classifier models.
- The metric surpasses all other metrics by large margins on M1, M2, M4, and M5.
- All other metrics fail to capture human correlation on caption detailness, while the proposed metric correlates reasonably well with humans on M4.
E. How to Use the Proposed Metric in Practice
The proposed metric should be evaluated under fixed discriminator and training settings with cross-validation folds. The accompanying materials provide exemplar caption scores and define the COCO comparison context.
- Fix the discriminator architecture, training hyperparameters, and training iterations before evaluating submissions.
- Train the discriminator on one test fold and evaluate the other, then reverse the folds to obtain results on the full test set.
- Figure 10 presents exemplar candidate captions with evaluation scores from the proposed metric and SPICE on the COCO validation set.
- Table 3 reports Pearson’s ρ correlations between human judgments and evaluation metrics using 12 entries from the 2015 MS-COCO captioning challenge.