PaperlayerView exact revision on arXiv

Source-linked AI summary

Show, Tell and Discriminate: Image Captioning by Self-retrieval with Partially Labeled Data

Xihui Liu, Hongsheng Li, Jing Shao, Dapeng Chen, Xiaogang Wang

arXiv:1803.08314v3cs.CV

TL;DR

Conventional captioners often repeat templated phrases and miss image-specific distinctions. This paper trains caption generation with self-retrieval guidance, reporting more discriminative captions and state-of-the-art performance on COCO and Flickr30k.

  • Problem

    Conventional captioning methods tend to repeat training-set phrases and produce generic captions that miss differences between images.

    Source: §1 · pp. 1–2

  • Method

    A self-retrieval module rewards captions for retrieving their corresponding images and guides caption generation, including training with partially labeled data.

    Source: §1 · p. 2

  • Results

    The approach achieves state-of-the-art performance on COCO and Flickr30k, with more discriminative captions and higher self-retrieval performance than conventional methods.

    Source: §1 · p. 3, §5 · p. 14

  • Takeaways & Limitations

    The framework can use additional unlabeled images without laborious annotations, and moderately hard negatives further boost fidelity and discriminativeness.

    Source: §1 · p. 2

Abstract

from arXiv · show

The aim of image captioning is to generate captions by machine to describe image contents. Despite many efforts, generating discriminative captions for images remains non-trivial. Most traditional approaches imitate the language structure patterns, thus tend to fall into a stereotype of replicating frequent phrases or sentences and neglect unique aspects of each image. In this work, we propose an image captioning framework with a self-retrieval module as training guidance, which encourages generating discriminative captions. It brings unique advantages: (1) the self-retrieval guidance can act as a metric and an evaluator of caption discriminativeness to assure the quality of generated captions. (2) The correspondence between generated captions and images are naturally incorporated in the generation process without human annotations, and hence our approach could utilize a large amount of unlabeled images to boost captioning performance with no additional laborious annotations. We demonstrate the effectiveness of the proposed retrieval-guided method on COCO and Flickr30k captioning datasets, and show its superior captioning performance with more discriminative captions.

1 Introduction

Conventional captioners often produce templated descriptions that miss image-specific differences. The paper uses self-retrieval guidance to encourage discriminative captions and makes use of unlabeled images during training.

1 Introduction

  • Conventional methods often repeat training-set phrases, producing generic captions that miss distinctions between images, such as a bird standing on a mirror.About fifty percent of conventionally generated captions exactly match training-set captions.Source: §1 · pp. 1–2
  • A self-retrieval module evaluates captions by whether they retrieve their corresponding images, and its reward guides caption generation through REINFORCE.The framework aims to improve both caption discriminativeness and fidelity.Source: §1 · p. 2
  • For unlabeled images, the framework uses self-retrieval rewards without requiring laborious annotations; labeled images also use ground-truth captions in reward calculation.The framework can therefore use additional unlabeled images during training.Source: §1 · p. 2
  • Mining moderately hard negative samples from unlabeled data further boosts caption fidelity and discriminativeness.Source: §1 · p. 3
  • The approach achieves state-of-the-art performance on COCO and Flickr30k, with more discriminative captions and higher self-retrieval performance than conventional methods.Additional unlabeled data could further boost captioning performance.Source: §1 · p. 3

2 Related Work

Prior captioning work includes template, search, and encoder-decoder methods, alongside reinforcement learning and approaches targeting caption diversity. This paper uses self-retrieval to encourage discriminative captions and distinguishes its approach through unlabeled-image training and hard-negative mining.

2 Related Work

  • Captioning research includes template-based, search-based, and language-based encoder-decoder methods, with attention and high-level attributes also used to aid captioning.Source: §2 · p. 3
  • Maximum-likelihood training can cause exposure bias and mismatch evaluation objectives, while reinforcement-learning techniques address these issues and improve captioning performance.Source: §2 · p. 3
  • Vedantam et al. introduced discriminative context-aware captioning with generic context-agnostic training, but inference requires a distractor class or image, which is difficult to obtain in generic captioning applications.Recent works [40] aim to increase caption diversity; GANs can generate diverse, human-like captions.Source: §2 · p. 3
  • The proposed self-retrieval module evaluates generated captions through text-to-image retrieval; unlike concurrent work, the approach also uses unlabeled images and moderately hard negatives.The figure description specifies a shared image encoder for captioning and retrieval, with retrieval rewards back-propagated using REINFORCE.Source: §2 · pp. 3–4, Figure 2 · p. 4

3 Methodology

The framework trains a captioner to produce descriptions that retrieve their source images over distractors, then extends retrieval guidance to unlabeled images. Training combines supervised caption fidelity with retrieval discrimination where annotations are available and uses staged optimization.

3 Methodology

  • A CNN–LSTM captioner is paired with a text-to-image self-retrieval module that evaluates whether each generated caption matches its source image over distractors.The captioning module encodes images and decodes visual features into captions; the retrieval module compares generated captions with corresponding and other images.Source: §3 · p. 4, §3.1 · p. 5

3.1 Image Captioning with Self-retrieval Reward

  • The self-retrieval loss requires a caption’s matching image to exceed its hardest distractor’s similarity by margin m, while CIDEr and retrieval rewards balance fidelity and discrimination.The retrieval module serves as a discriminativeness evaluator, and the reward is optimized through sampled captions.Source: §3.1 · p. 6, Back-propagation by REINFORCE algorithm · p. 7

3.2 Improving Captioning with Partially Labeled Images

3.3 Training Strategy

  • Training pretrains the captioner with cross-entropy, then applies REINFORCE with CIDEr plus retrieval rewards for labeled images and retrieval-only rewards for unlabeled images.Unlabeled images in each mini-batch are selected by moderately hard negative mining.Source: §3.3 · p. 9

4 Experiments

Experiments evaluate the framework on COCO and Flickr30k with standard caption metrics and test both fully and partially labeled training. The results and analyses report gains from self-retrieval, unlabeled data, and discriminative captions.

4.1 Datasets and Evaluation Criteria

  • The evaluation uses COCO and Flickr30k, the COCO Karpathy split, and BLEU, METEOR, ROUGE-L, CIDEr-D, and SPICE.The COCO split uses 5,000 validation images, 5,000 test images, and 82,783 training images.Source: §4.1 · p. 9

4.2 Implementation Details

  • The captioner shares ResNet-101 features with fixed retrieval components, uses top-down attention and language LSTMs, and generates captions with beam size 5.Training pretrains with cross-entropy loss and then uses REINFORCE; partially labeled mini-batches use a 1:1 labeled-to-unlabeled proportion.Source: Captioning module · p. 10

4.3 Results

  • The self-retrieval framework improves most metrics over the baseline with fully labeled data, and additional unlabeled data further enhances performance.The comparisons cover COCO and Flickr30k; SR-PL uses both labeled and unlabeled images.Source: §4.3 · p. 10, Table 1 · p. 11, Table 2 · p. 11
  • Qualitatively, the proposed model distinguishes similar images with details such as flower color and vase type, where the baseline repeats generic captions.Examples include “red flowers,” “white vase,” and “in a garden.”Source: §4.3 · p. 11, Figure 4 · p. 12

4.5 Discriminativeness of Generated Captions

  • On COCO’s Karpathy test split, generated captions improve text-to-image retrieval and significantly increase caption uniqueness and novelty.Retrieval recall measures whether the corresponding image appears among the top-k results for a generated caption used as a query.Source: Table 4 · p. 13, §4.5 · p. 14

4.4 Ablation Study

  • VSE++, VSE0, and softmax retrieval losses all outperform the baseline, with VSE++ slightly better among the tested formulations.The three formulations are compared in the COCO ablation study.Source: §4.4 · pp. 12–13
  • α = 1 and a 1:1 labeled/unlabeled mini-batch perform best; hard-negative indices [100, 1000] work best, while the hardest negatives can confuse matching.Excessive self-retrieval reward harms performance because it fails to optimize CIDEr; a caption may also naturally correspond to multiple images.Source: §4.4 · p. 13

5 Conclusions

The framework addresses templated, generic captions by training a captioning module with self-retrieval reward. It produces more discriminative captions and achieves state-of-the-art performance on two widely used image-captioning datasets.

5 Conclusions

  • The framework targets captions that conventional approaches make templated and generic.It explicitly improves caption discriminativeness through training with self-retrieval reward.Source: §5 · p. 14
  • A captioning module receives self-retrieval reward through the REINFORCE algorithm to boost generated-caption discriminativeness.Source: §5 · p. 14
  • The framework yields more discriminative captions and state-of-the-art performance on two widely used image-captioning datasets.Source: §5 · p. 14
Loading 1803.08314v3…