PaperlayerView exact revision on arXiv

Source-linked AI summary

Fine-grained Image Captioning with CLIP Reward

Jaemin Cho, Seunghyun Yoon, Ajinkya Kale, Franck Dernoncourt, Trung Bui, Mohit Bansal

arXiv:2205.13115v2cs.CLcs.AIcs.CV

TL;DR

Text-similarity objectives can favor prominent objects in reference captions and leave out fine details that distinguish images. The paper uses CLIP image-text similarity as a caption reward and finetunes its text encoder for grammar, then reports more distinctive captions and stronger fine-grained evaluation outcomes. The authors frame deployed use as out of scope because the web image-text data used to train CLIP can contain problematic pairs.

  • Problem

    Reference captions in public datasets often describe prominent objects, and text-similarity-trained models can omit fine details that distinguish images.

    Source: §1 · p. 1

  • Method

    The method rewards CLIP image-text similarity, finetunes CLIP's text encoder with synthetic negative captions to improve grammar, and introduces FineCapEval for fine-grained caption evaluation.

    Source: §1 · p. 1, §3.2 · p. 3

  • Results

    CLIP-guided captions are more distinctive and detailed than CIDEr-optimized captions, exceed paired reference captions on text-to-image retrieval, and outperform text-similarity objectives across FineCapEval categories and human analysis.

    Source: §1 · pp. 1–2

  • Takeaways & Limitations

    CLIP reward enables caption training without reference captions in reward computation, while grammar finetuning addresses repetition artifacts from the naive CLIP reward.

    Source: §1 · p. 2, §7 · p. 5

    The authors state that CLIP-reward captioning is intended as a research output and deployed use is out of scope, given problematic image-text pairs in its web training data.

    Source: §8 · p. 5

Abstract

from arXiv · show

Modern image captioning models are usually trained with text similarity objectives. However, since reference captions in public datasets often describe the most salient common objects, models trained with text similarity objectives tend to ignore specific and detailed aspects of an image that distinguish it from others. Toward more descriptive and distinctive caption generation, we propose using CLIP, a multimodal encoder trained on huge image-text pairs from web, to calculate multimodal similarity and use it as a reward function. We also propose a simple finetuning strategy of the CLIP text encoder to improve grammar that does not require extra text annotation. This completely eliminates the need for reference captions during the reward computation. To comprehensively evaluate descriptive captions, we introduce FineCapEval, a new dataset for caption evaluation with fine-grained criteria: overall, background, object, relations. In our experiments on text-to-image retrieval and FineCapEval, the proposed CLIP-guided model generates more distinctive captions than the CIDEr-optimized model. We also show that our unsupervised grammar finetuning of the CLIP text encoder alleviates the degeneration problem of the naive CLIP reward. Lastly, we show human analysis where the annotators strongly prefer the CLIP reward to the CIDEr and MLE objectives according to various criteria. Code and Data: https://github.com/j-min/CLIP-Caption-Reward

1 Introduction

Detailed, distinguishing image descriptions matter for image search and accessibility, but reference-caption similarity can favor prominent objects over fine-grained details. The paper proposes CLIP-based rewards and grammar finetuning, and reports more distinctive captions and stronger fine-grained evaluation results than text-similarity objectives.

1 Introduction

  • Text-similarity training can favor prominent objects in public-dataset references and produce captions that omit fine details distinguishing an image from others.Detailed descriptions are important for image-search keys and accessibility for visually impaired people.Source: §1 · p. 1
  • The proposed reward uses CLIP image-text similarity, while CLIP text-encoder finetuning improves grammar without extra text annotations and removes references from reward computation.The paper also introduces FineCapEval, covering overall, background, object, and object-relation criteria.Source: §1 · p. 1
  • CLIP-guided captions are more distinctive and detailed than CIDEr-optimized captions, outperform paired reference captions on text-to-image retrieval, and outperform text-similarity objectives across FineCapEval categories and human analysis.The paper also reports that CLIP text-encoder finetuning improves grammar by removing artifacts such as word repetition.Source: §1 · pp. 1–2

2 Related Works

Caption evaluation based on n-grams or scene graphs can miss paraphrases when references or scene graphs are limited. Related work explores pretrained language and multimodal relevance scores, alternative reward signals, and the balance needed when combining retrieval and text-similarity rewards.

2 Related Works

  • N-gram and scene-graph metrics can miss paraphrased expressions when reference captions or scene graphs are limited, motivating pretrained language- or multimodal-model relevance scores.Examples include BERTScore, ViLBERTScore, UMIC, and CLIPScore.Source: §2 · p. 2
  • Prior captioning rewards include visual-question-answering accuracy and image-text self-retrieval scores, with retrieval approaches encouraging captions distinctive to each image.Some retrieval approaches combine self-retrieval with n-gram metrics.Source: §2 · p. 2
  • Prior self-retrieval methods require balancing retrieval and text-similarity objectives for stable training, whereas the proposed approach removes reference captions and text-similarity metrics from reward computation.Figure 2 contrasts reference-caption text-similarity rewards such as CIDEr with the proposed CLIP image-text reward.Source: §2 · p. 2, Figure 2 · p. 2

3 Methods

The method trains captioning models with CLIP image-text similarity as a reward and finetunes CLIP's text encoder with synthetic negative captions to improve grammar. The resulting reward combines image-text similarity with a grammar score.

3.1 CLIP-guided Image Captioning

  • CLIP-S rewards image-caption similarity using CLIP's image and text encoders, encouraging captions with distinctive information about the input image.The stated weight is w = 2.5; captioning-model optimization uses REINFORCE with a self-critical baseline, comparing beam-search reward against greedy-decoding reward.Source: §3.1 · pp. 2–3

3.2 Improving Grammar with CLIP Text Encoder Finetuning

  • Because CLIP lacks a language-modeling objective, the method finetunes its text encoder and a grammar head using reference captions as positive examples and synthetically corrupted captions as negatives.The image encoder is fixed during finetuning, and the caption reward combines CLIP-S and grammar score as R(I, c) = CLIP-S(I, c) * λ + g(c), with λ = 2.Source: §3.2 · p. 3

4 FineCapEval: Fine-grained Caption Evaluation Dataset

FineCapEval is introduced to evaluate caption quality across overall description, background, objects, and relations between objects. It contains human-written annotations for images drawn from two datasets.

4 FineCapEval: Fine-grained Caption Evaluation Dataset

  • FineCapEval contains 1,000 images and 5,000 annotations for each of four criteria: background, objects and attributes, object relations, and a detailed overall caption.The images are drawn equally from the MS COCO test2015 split and Conceptual Caption validation split, with five annotators writing phrases for each image.Source: §4 · p. 3

5 Experiments

Experiments compare captioning rewards on MS COCO using text-similarity, image-text, retrieval, and fine-grained evaluation measures, alongside human preference judgments.

5 Experiments

  • Experiments compare MLE, CIDEr, CLIP-S, CIDEr+CLIP-S, and CLIP-S+Grammar rewards on the MS COCO Karpathy split.The evaluation also includes human judgments of captions on five criteria.Source: §5 · pp. 3–4
  • The captioning model uses CLIP-Res50 visual features and a six-layer transformer encoder-decoder, trained first with MLE and then with reward objectives.Images are resized to 224×224; training uses 15 MLE epochs followed by 25 reward-training epochs.Source: §5 · pp. 3–4
  • Evaluation reports BLEU-4, CIDEr, METEOR, ROUGE-L, BERT-S, CLIP-S, and RefCLIP-S, which measure reference-text or image-text similarity.RefCLIP-S averages textual similarity to references and image-text similarity to the input image.Source: §5 · p. 4
  • Text-to-image retrieval measures recall of the reference image, while FineCapEval uses CIDEr for overall captions and word-level recall for background, object, and relation criteria.Retrieval uses CLIP ViT-B/32; retrieval scores use the first of five reference captions.Source: §5 · p. 4, Table 2 · p. 4
  • Human evaluation uses pairwise caption choices across overall, background, object, attribute, and relation criteria, with 10 annotators answering 50 questions per criterion.The comparison uses 50 images from the FineCapEval set.Source: §5 · p. 4

6 Results and Discussions

CLIP-guided rewards yield more distinctive captions, while grammar finetuning mitigates repetitive output and improves text-similarity scores. CLIP rewards also outperform CIDEr on FineCapEval, and human annotators prefer CLIP-S+Grammar captions across criteria.

6.1 CLIP Guides Distinctive Captions

  • CLIP-S and CLIP-S+Grammar achieve higher image-text metrics and text-to-image retrieval scores than baselines, even exceeding retrieval scores for reference captions.CLIP-guided captions can include details absent from references, such as a restaurant's “blue sign,” despite scoring lower on reference-based text metrics than CIDEr.Source: §6.1 · p. 4

6.2 Finetuning CLIP Text Encoder Improves Grammar

  • Adding the grammar reward mitigates word-repetition degeneration in CLIP-S captions and increases all text-similarity metrics, including CIDEr by 60 points.The qualitative examples also show CLIP-S+Grammar describing rainy weather as “wet.”Source: §6.2 · pp. 4–5, Table 3 · p. 5

6.3 Fine-grained Caption Evaluation

  • CLIP-S and CLIP-S+Grammar outperform CIDEr on all four FineCapEval criteria, with the smallest gap for object information.Human annotators also strongly prefer CLIP-S+Grammar captions over CIDEr and MLE captions on all evaluated criteria.Source: §6.3 · p. 5

7 Conclusion and Future Directions

The paper proposes CLIP-similarity reward training with text-encoder finetuning for grammar, and introduces FineCapEval for fine-grained caption evaluation. Its future directions include style-specific reward finetuning and improved synthetic augmentation.

7 Conclusion and Future Directions

  • The proposed training strategy maximizes CLIP multimodal similarity and finetunes its text encoder for grammar, eliminating reference captions from reward computation.The paper also introduces FineCapEval and reports improvements in retrieval, fine-grained evaluation, and human evaluation.Source: §7 · p. 5
  • Future work includes finetuning CLIP reward models for desired writing styles and improving synthetic augmentation with external data and advanced linguistic expertise.Source: §7 · p. 5

8 Ethical Considerations

The paper flags ethical limits in the use of CLIP-trained captioning models and describes how its reward variants differ in example captions.

8 Ethical Considerations

  • CLIP models use millions of web image-text pairs that may include explicit and problematic content, and deployed use of the captioning models is out of scope.Source: §8 · p. 5
  • The CLIP and captioning models are trained on English datasets, and their use is limited to English-language cases.Source: §8 · pp. 5–6

A More Image Captioning Examples

B Implementation Details

The implementation uses synthetic negative captions for grammar finetuning and reports evaluation scores from a single model initialization.

  • Grammar finetuning generates negative captions by repeating, removing, inserting, swapping, or shuffling words in original captions.Source: B Implementation Details · p. 7
  • MS COCO caption metrics including CIDEr are computed with pycocoevalcap, while BERTScore uses the official repository with roberta-large.Source: B Implementation Details · p. 7
  • Evaluation scores are reported from a single weight initialization because initial experiments showed no meaningful fluctuation across runs.Source: B Implementation Details · p. 7

C FineCapEval Details

FineCapEval data collection elicits detailed English image descriptions and metadata for background, objects and attributes, and relationships.

  • Annotators write captions covering image background, objects and attributes, and relationships, and mark which words or phrases express those criteria.Source: C FineCapEval Details · p. 7
  • Captions are constrained to one sentence of at least ten words, avoid subjective subject descriptions and named entities, and attend to local and global events.Source: C FineCapEval Details · p. 7
  • The collected captions are in English and written by native English speakers living in the United States; quality assurance manually examines 50% of captions.Source: C FineCapEval Details · pp. 7–8
  • Word-level recall measures the average proportion of words from reference phrases that appear in generated captions.Source: C FineCapEval Details · p. 8

D Human Evaluation Details

Human evaluation compares captions across fine-grained criteria, while the appendix also records the licenses for the referenced datasets and models.

D Human Evaluation Details

  • For each criterion, annotators compare the paper's CLIP-S+Grammar caption with an MLE or CIDEr baseline and select the one that better describes the image.The evaluation covers overall, background, object, attribute, and relation criteria, with a tie option when captions are equally good or bad.Source: D Human Evaluation Details · pp. 8, 11
  • The negative-caption generator randomly applies repetition, removal, insertion, swapping, or shuffling to tokens.Source: D Human Evaluation Details · p. 9
  • Word-level recall averages, across reference phrases, the fraction of phrase words found in a generated caption.Source: D Human Evaluation Details · p. 10
  • For each criterion, the study recruits 10 Amazon Mechanical Turk annotators located in Great Britain or the United States, with HIT approval rates above 80% and more than 1,000 approved HITs.Source: D Human Evaluation Details · p. 11

E Licenses

  • The listed licenses include CC 4.0 for MS COCO and MIT for CLIP and CLIP-ViL.Source: E Licenses · p. 11

Related papers

Loading 2205.13115v2…