PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Unsupervised Image Captioning with Shared Multimodal Embeddings

Iro Laina, Christian Rupprecht, Nassir Navab

arXiv:1908.09317v1cs.CV

TL;DR

Image captioning has traditionally depended on annotated image-caption pairs, despite the availability of independent image and language sources. This paper aligns image features with semantically structured text embeddings using visual concepts, robust alignment, and conditional adversarial training. It reports state-of-the-art performance in unpaired COCO captioning and captions that can extend beyond detector labels.

  • Problem

    Traditional image captioning relies on human-annotated image-caption pairs, while unsupervised generation from independent image and language sources remains mostly unexplored.

    Source: §1 · p. 1

  • Method

    The method learns a semantic text embedding space and aligns image features to it using visual concepts, robust training, and a conditional adversarial component.

    Source: §3 · p. 3, §3.3 · p. 5

  • Results

    The method sets a new state of the art for unpaired COCO captioning and can generate meaningful captions with concepts beyond the detector’s fixed labels.

    Source: §1 · p. 2, §4.1 · p. 6

  • Takeaways & Limitations

    The approach supports captioning with varied image and language sources, including language sources in another language or with particular styles.

    Source: §1 · p. 2, §6 · p. 8

    Initial assignments require the language source to contain sufficient visual concepts that overlap with the image domain.

    Source: §5 · p. 8

Abstract

from arXiv · show

Understanding images without explicit supervision has become an important problem in computer vision. In this paper, we address image captioning by generating language descriptions of scenes without learning from annotated pairs of images and their captions. The core component of our approach is a shared latent space that is structured by visual concepts. In this space, the two modalities should be indistinguishable. A language model is first trained to encode sentences into semantically structured embeddings. Image features that are translated into this embedding space can be decoded into descriptions through the same language model, similarly to sentence embeddings. This translation is learned from weakly paired images and text using a loss robust to noisy assignments and a conditional adversarial component. Our approach allows to exploit large text corpora outside the annotated distributions of image/caption data. Our experiments show that the proposed domain alignment learns a semantically meaningful representation which outperforms previous work.

1. Introduction

Image captioning traditionally relies on costly human-annotated image-caption pairs, while benchmarks cover limited object categories and may not transfer readily to real-world settings. This work explores captioning from independent image and language sources, aligning images to semantically structured text embeddings through visual concepts.

1. Introduction

  • Traditional captioning uses human-annotated image-caption pairs, whose collection requires extensive quality control and manual cleaning.The captions also tend to be short and repetitive, while benchmarks cover limited object categories and use imperfect evaluation metrics.Source: §1 · p. 1
  • Figure 1 depicts disjoint image and text sources whose features are mapped into a shared, concept-structured space, from which image captions are decoded at inference.The learned space reflects visual concepts and their co-occurrence.Source: Figure 1 · p. 1
  • Unsupervised captioning uses independent image and language sources, offering access to large unlabeled or weakly labeled image collections and text corpora without costly annotations.The task of unsupervised image-description generation remains mostly unexplored in the passage’s account.Source: §1 · pp. 1–2
  • The method learns semantically structured sentence embeddings, then weakly aligns image features to them using visual concepts and cross-domain co-occurrence statistics.For example, the text-domain association between boat and water parallels their co-occurrence in images.Source: §1 · p. 2
  • A pretrained object detector supplies an initial noisy alignment between the text source and visual entities detected in images.This provides weak supervisory signal when images and language come from different sources.Source: §1 · p. 2
  • The model predicts meaningful captions that can include concepts beyond the detector’s fixed labels, nearly matches some early supervised methods, and outperforms previous unsupervised methods.Visual concept co-occurrence can support concepts such as beach even when they are absent from the detector’s label set.Source: §1 · p. 2

2. Related Work

Related work spans fully supervised captioning, novel-object description, partially supervised settings, and multimodal embeddings. The paper distinguishes its approach from adaptation methods that start with a fully supervised captioner and instead aligns compatible but disjoint image and language sources.

Fully supervised

  • Fully supervised captioning commonly uses a CNN image encoder and RNN language decoder, with later work advancing attention, visual grounding, video and dense captioning, attributes, relationships, diversity, style, and bias mitigation.The cited work also explores optimizing evaluation metrics.Source: §2 · p. 2

Novel object captioning

  • Novel-object captioning uses few labeled pairs, copying mechanisms, independent image and text sources, or detected-concept templates to incorporate entities absent from paired training data.Another approach constrains beam search to target words.Source: §2 · p. 2

Partial supervision

  • Partial-supervision approaches include cross-domain adaptation, stylized descriptions, sequence completion, and translation through an intermediary language, but typically begin with a fully supervised captioner.The paper distinguishes its goal of exploiting disjoint yet compatible image and language sources from aligning a target domain to a supervised source domain.Source: §2 · p. 2

Multimodal embeddings

  • Multimodal embeddings align latent representations across modalities; prior uses include translation, captioning, cross-modal retrieval, video captioning, and ranking caption candidates.Some joint spaces also support vector arithmetic.Source: §2 · p. 3

3. Methods

The method builds a visually structured sentence-embedding space, then aligns image features to it from weak image-text correspondences so the shared decoder can produce captions.

3.1. Language Model

  • A seq2seq language model is trained on text, with visual-concept triplets pulling sentences sharing at least two concepts together and pushing concept-disjoint sentences apart.Pairs sharing only one concept are ignored to reduce bad alignments.Source: §3.1 · p. 3

3.2. Joint Image and Language Domain

  • Images and sentences are weakly paired in a bipartite graph weighted by overlapping concepts, with image-side hyponyms broadening matching beyond detector labels.Pairs without concept overlap are excluded from training.Source: §3.2 · p. 4

3.3. Learning the Semantic Alignment

  • The image-to-text mapping uses robust alignment toward real sentence embeddings and a conditional adversarial discriminator that checks image-concept/text-feature consistency.The robust loss approximates its minimum over a fixed number K of sampled sentence embeddings per image.Source: §3.3 · p. 5

4. Experiments and Results

Experiments evaluate the method in unpaired and unsupervised settings, then examine the learned joint embedding. The full unpaired model sets a new state of the art, while external text domains support captioning and question generation with distinct qualitative trade-offs.

4.1. Unpaired Captioning

  • Robust alignment, semantic sentence embeddings, and conditional adversarial training each improve performance; the naive L2 alignment does not improve over MLE alone.The ablations use COCO object categories for initial alignment; the robust loss uses K = 10 candidate sentences per image.Source: Table 1 · p. 6, §4.1 · p. 6
  • Without jointly finetuning the decoder, alignment-only captions are grammatically incoherent; weak assignments are noisy, and MLE-only training can repeat sub-phrases.The oracle evaluation also scores generally low because the initial assignments are very noisy.Source: §4.1 · p. 6
  • The full model sets a new state of the art on COCO under the unpaired setup, using OpenImages categories for concept extraction.The reported model uses K = 10 candidate samples and beam size 3.Source: §4.1 · p. 6, Table 2 · p. 8

4.2. Unsupervised Captioning

  • With independent image and text sources, adversarial training consistently improves performance, while GCC-based assignments yield short, simple descriptions but can capture distinctions such as a plane on the ground versus in the air.Using a different language domain also changes vocabulary and style relative to COCO ground-truth captions.Source: §4.2 · p. 7
  • Using VQA-v2 questions as the text source leads the model to ask questions about image content rather than produce captions.The experiments also pair Flickr30k images with COCO captions and COCO images with GCC sentences.Source: §4.2 · p. 7

4.3. Joint Embedding Visualization

  • The joint t-SNE embedding is organized by visual categories, with image-projected and sentence features mixed within clusters.The authors interpret the mixing as evidence that the modalities are not separable in the learned space.Source: §4.3 · p. 7

5. Limitations and Discussion

The approach depends on overlap between text-source concepts and the image domain, and its ability to describe novel compositions is limited by the base captioner and the available visual concepts. Relationship detection could help resolve ambiguities, but conflicts with weak or no supervision.

5. Limitations and Discussion

  • The text source must contain sufficient visual concepts overlapping with the image domain to generate initial assignments.The authors suggest combining text sources with varying contents and styles as a possible mitigation.Source: §5 · p. 8
  • The base captioner limits extension to novel compositions and atypical scenes; about 20% of generated captions are unique and 16% are novel, matching [60].Unsupervised training will not solve limitations already present in the supervised model [60].Source: §5 · p. 8
  • Discovering distinctions such as whole pizza versus a slice requires the concept “slice” to appear in another context; more concepts may enable greater diversity.Predicates in the weak-assignment concept set could support relationship detection, but this goes against weak or no supervision.Source: §5 · p. 8
  • Figure 4 projects the learned joint embedding, where text sentences and image embeddings lie in visual-semantic groups colored by visual-concept group.The figure zooms into a cluster to show the grouping.Source: Figure 4 · p. 8
  • Relationship detection could help distinguish a person riding a bike from carrying one, but it conflicts with the weak- or no-supervision premise.Source: §5 · p. 8

6. Conclusion

The method aligns images and text in a visual-concept-structured shared latent representation using noisy initial correspondences, robust training, and adversarial image-to-text translation. Experiments improve the state of the art in unpaired COCO captioning, while future work could explore decoder changes to encourage novel compositions.

6. Conclusion

  • The experiments improve the state of the art in the unpaired COCO setting.Source: §6 · p. 8
  • The method aligns images and text in a shared latent representation structured through visual concepts, using a pretrained recognition model for noisy initial correspondences.Robust training and adversarial learning of image-feature translation to text allow the model to learn captioning.Source: §6 · p. 8
  • Future work could improve the decoder with attention or use a template approach to encourage novel object compositions.Source: §6 · p. 8

Related papers

Loading 1908.09317v1…