Source-linked AI summary
Unsupervised Image Captioning
Yang Feng, Lin Ma, Wei Liu, Jiebo Luo
TL;DR
Paired image-sentence datasets are expensive to acquire, so the paper investigates image captioning without labeled pairs. It trains a model from independent images, an external sentence corpus, and a visual concept detector using adversarial, concept, and reconstruction objectives. The method produces promising captions, achieving CIDEr 28.6% on MSCOCO, while relying on a challenging cross-modal alignment setting.
Problem
Paired image-sentence datasets are expensive to acquire, and fully unsupervised captioning from independent images and sentences remains challenging.
Method
The model uses an image set, external sentence corpus, visual concept detector, adversarial sentence generation, concept rewards, and bi-directional image-sentence reconstruction.
Results
CIDEr 28.6% on MSCOCO was achieved with the proposed unsupervised training method, compared with 22.9% when trained from scratch without initialization.
Takeaways & Limitations
The paper demonstrates promising unsupervised image captioning without paired caption annotations, supported by a corpus of over two million image-description sentences.
Takeaways & Limitations
The task is especially challenging because images and sentences reside in two modalities with significantly different characteristics.
Abstract
from arXiv · showhide
Deep neural networks have achieved great successes on the image captioning task. However, most of the existing models depend heavily on paired image-sentence datasets, which are very expensive to acquire. In this paper, we make the first attempt to train an image captioning model in an unsupervised manner. Instead of relying on manually labeled image-sentence pairs, our proposed model merely requires an image set, a sentence corpus, and an existing visual concept detector. The sentence corpus is used to teach the captioning model how to generate plausible sentences. Meanwhile, the knowledge in the visual concept detector is distilled into the captioning model to guide the model to recognize the visual concepts in an image. In order to further encourage the generated captions to be semantically consistent with the image, the image and caption are projected into a common latent space so that they can reconstruct each other. Given that the existing sentence corpora are mainly designed for linguistic research and are thus with little reference to image contents, we crawl a large-scale image description corpus of two million natural sentences to facilitate the unsupervised image captioning scenario. Experimental results show that our proposed model is able to produce quite promising results without any caption annotations.
1. Introduction
Existing captioning methods rely heavily on paired image-sentence data, motivating unsupervised training from independent images and sentences. The proposed approach combines adversarial sentence generation, visual concept distillation, reconstruction, unlabeled-data initialization, and a large external corpus.
- Paired image-sentence datasets are labor-intensive to acquire, relatively small, and limited in object-category variety, restricting generalization to images in the wild.
- Prior approaches reduce paired-data dependence through novel-object captioning, cross-domain transfer, pivot-language captioning, or semi-supervised learning, but do not provide fully unsupervised captioning.
- The paper introduces unsupervised image captioning using an image set, an external sentence corpus, and an existing visual concept detector, without labeled image-sentence pairs.
- Adversarial training teaches plausible sentence generation, while concept rewards encourage generated words to reflect visual concepts detected in the image.
- Image-sentence reconstruction projects both modalities into a common latent space so generated captions remain semantically consistent with images.
- A concept-to-sentence initialization pipeline creates pseudo image-sentence pairs, and the authors crawl over two million Web sentences for training.
2. Related Work
Related work includes supervised and partially or indirectly supervised captioning methods, alongside unsupervised machine translation. The paper distinguishes its setting because images and sentences occupy substantially different modalities.
- 2.1. Image Captioning: Supervised captioning commonly uses a CNN to encode images and an RNN to generate sentences from paired image-caption data.
- 2.1. Image Captioning: The proposed model architecture contains an image encoder, sentence generator, and sentence discriminator.
- 2.1. Image Captioning: Partial-supervision methods use incomplete training sequences represented by finite state automata to reduce annotation requirements.
- 2.2. Unsupervised Machine Translation: Unsupervised machine translation maps source and target languages into a common latent space, whereas image captioning must align images and sentences across different modalities.
3. Unsupervised Image Captioning
The model trains image captioning without paired image-sentence data by combining image encoding, sentence generation, and discrimination with three complementary objectives. Adversarial generation promotes plausible sentences, concept distillation adds image relevance, and bidirectional reconstruction encourages semantic consistency.
- 3.1. The Model: The model uses an image encoder, an LSTM sentence generator, and an LSTM discriminator to train with unpaired images and sentences.The image encoder produces image features; the generator decodes them into sentences, while the discriminator distinguishes generated partial sentences from corpus sentences.
- 3.2.1 Adversarial Caption Generation: Adversarial training rewards generated sentences that the discriminator considers increasingly similar to real sentences from the external corpus.The generator receives an adversarial reward at each time step based on the discriminator’s estimated probability that the generated text is real.
- 3.2.2 Visual Concept Distillation: Concept distillation rewards words corresponding to visual concepts detected in the input image, using each concept detector confidence score.The concept detector outputs concept-confidence pairs, and the matching reward is assigned to generated words through an indicator function.
- 3.2.3 Bi-directional Image-Sentence Reconstruction: The model projects images and sentences into a common latent space so they can reconstruct one another and encourage semantic image-caption consistency.Image reconstruction uses generated sentences to reconstruct image features rather than full high-resolution images, while sentence reconstruction decodes sentences from projected representations.
- 3.2.4 Integration: The three objectives are jointly optimized, with policy-gradient updates for the generator and gradient-descent updates for the discriminator.The generator combines adversarial, concept, and reconstruction rewards with sentence-reconstruction gradients, and the generator and discriminator are updated alternately.
- 3.3. Initialization: An initialization pipeline pre-trains the generator and discriminator because training the full model from scratch on unlabeled data is challenging.The discriminator is initialized using adversarial sentence generation on the sentence corpus.
4. Experiments
Experiments evaluate unsupervised captioning on MSCOCO using unpaired images, a crawled Shutterstock sentence corpus, and an OpenImages-trained concept detector. Results show benefits from initialization and combining adversarial, concept, and reconstruction objectives.
- Experimental Settings: The experiments use MSCOCO images without captions for training, Shutterstock descriptions as the sentence corpus, and an OpenImages-trained detector for visual concepts.The MSCOCO captions are reserved for evaluation, while the crawled descriptions are tokenized and filtered into the training corpus.
- Performance Results: 28.6% CIDEr is achieved by the proposed unsupervised method on the MSCOCO test split, compared with 22.9% when training from scratch without initialization.The authors attribute the difference to the initialization pipeline benefiting model training.
- Ablation Studies: 22.5% and 23.5% CIDEr are obtained by “con2sen” and “feat2sen”, respectively, supporting their use for pseudo-pair generation and generator initialization.The ablation identifies “con2sen” as a source of pseudo image-caption pairs and “feat2sen” as a meaningful generator initialization.
- Ablation Studies: The adversarial objective alone performs much worse because it encourages genuine sentences without ensuring semantic correlation with image content.Adding concept and image reconstruction objectives still leaves evaluation affected by linguistic differences between Shutterstock descriptions and COCO captions, whereas combining all objectives substantially improves performance.
- Qualitative Results: The full model can express visual attributes unavailable from the detector and correct erroneous detected concepts in qualitative examples.The reconstruction objective helps recognize a cat’s color, while the full objectives correct erroneous “vehicle” and “hat” detections.
- Effect of Concept Reward: The averaged number of correct concept words reaches about 0.8 for “adv + con”, “adv + con + im”, and “Ours”, versus about 0.6 for “Ours w/o init”.The authors associate this improvement with the concept reward and, for the initialized models, a better starting point.
- Unpaired Captioning: 54.9% CIDEr is reached under the unpaired captioning setting, increasing from 49% after adding sentence reconstruction to the other objectives.The three objectives improve performance step by step, while bidirectional image and sentence reconstruction encourages semantic consistency with images.
5. Conclusions
The paper presents an unsupervised image captioning method that avoids paired image-sentence data by combining three training objectives and a large-scale image description corpus. Experiments show promising results without labeled image-sentence pairs.
- The proposed method trains an image captioning model without using paired image-sentence data.
- Three objectives make generated captions resemble corpus sentences, convey image object information, and align image-sentence features through bidirectional reconstruction.
- A Shutterstock image description corpus containing over two million sentences was collected to support unsupervised image captioning.
- Experiments demonstrate quite promising results without using labeled image-sentence pairs.
- The authors identify human evaluation of unsupervised image captioning as future work.