PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Compositional Captioning: Describing Novel Object Categories without Paired Training Data

Lisa Anne Hendricks, Subhashini Venugopalan, Marcus Rohrbach, Raymond Mooney, Kate Saenko, Trevor Darrell

arXiv:1511.05284v2cs.CVcs.CL

TL;DR

Caption models trained on paired image-sentence corpora cover a relatively small range of objects and lack compositional integration of new objects. DCC transfers knowledge between semantically related concepts using unpaired image and text data, and the authors report richer image and video descriptions beyond paired-corpus coverage.

  • Problem

    Paired image-caption datasets cover fewer objects than recognition datasets, and existing caption models lack compositional structures for integrating new objects without paired examples.

    Source: §1 · p. 1

  • Method

    DCC combines unpaired image and text data, paired-data training, and transfer between semantically related concepts to compose captions for objects absent from paired datasets.

    Source: §1 · pp. 1–2

  • Results

    DCC's quantitative and qualitative results demonstrate new-vocabulary integration in image and video descriptions and improvement over current deep caption models.

    Source: §1 · p. 2, §6 · p. 8

  • Takeaways & Limitations

    DCC provides rich descriptions of new objects that are not limited by the availability of paired image-sentence corpora.

    Source: §6 · p. 8

    DCC sometimes omits new objects, produces grammatically incorrect sentences, or hallucinates objects.

    Source: §5.2 · p. 8

Abstract

from arXiv · show

While recent deep neural network models have achieved promising results on the image captioning task, they rely largely on the availability of corpora with paired image and sentence captions to describe objects in context. In this work, we propose the Deep Compositional Captioner (DCC) to address the task of generating descriptions of novel objects which are not present in paired image-sentence datasets. Our method achieves this by leveraging large object recognition datasets and external text corpora and by transferring knowledge between semantically similar concepts. Current deep caption models can only describe objects contained in paired image-sentence corpora, despite the fact that they are pre-trained with large object recognition datasets, namely ImageNet. In contrast, our model can compose sentences that describe novel objects and their interactions with other objects. We demonstrate our model's ability to describe novel concepts by empirically evaluating its performance on MSCOCO and show qualitative results on ImageNet images of objects for which no paired image-caption data exist. Further, we extend our approach to generate descriptions of objects in video clips. Our results show that DCC has distinct advantages over existing image and video captioning approaches for generating descriptions of new objects in context.

1 UC Berkeley, 2 ICSI, Berkeley, 3 UT Austin, 4 UMass Lowell · 1. Introduction

DCC composes descriptions of objects absent from paired caption data by combining unpaired visual and textual resources with knowledge transferred from known concepts, and is evaluated on held-out MSCOCO objects, ImageNet 7k images, and YouTube clips.

1. Introduction

  • DCC generates descriptions of objects absent from paired image-sentence corpora by combining visual data and text corpora that do not provide paired captions.Source: §1 · p. 1
  • Caption models built on paired corpora such as MSCOCO [20] and Flickr30k cannot compose descriptions of objects absent from those data, despite recognition systems covering thousands of classes in datasets such as ImageNet.Source: §1 · p. 1
  • DCC composes captions for novel elements such as otters by relating them to familiar concepts such as animals and combining them with known linguistic expressions.Source: §1 · pp. 1–2
  • Its lexical classifier and language model can be trained independently on unpaired image and text data, then combined and jointly trained on paired image-sentence data.Source: §1 · p. 2
  • External text corpora relate novel objects to concepts seen in paired data, enabling two mechanisms for transferring knowledge from known to novel objects.Source: §1 · p. 2
  • The MSCOCO evaluation uses a training split that excludes certain objects to test DCC's ability to caption new objects.Source: §1 · p. 2
  • Qualitative results show DCC describing objects in ImageNet 7k that are absent from caption datasets.Source: §1 · p. 2
  • DCC is also evaluated on a collection of YouTube video clips to demonstrate its use in describing novel objects in videos.Source: §1 · p. 2

2. Related Work

Related work spans deep image captioning, zero-shot transfer, and methods for describing new objects in context. DCC is distinguished by generating full-sentence descriptions of concepts absent from paired image-sentence data.

Deep Captioning

  • Captioning work includes CNN-RNN and multimodal models [5] [36] [15] [16] [21], retrieval [4], and visual-concept classifiers [6] [37]; DCC modifies multimodal captioning for concepts absent from paired data.CNN-RNN models condition word prediction on image features and previous words, while multimodal models embed image and language features; retrieval is limited to descriptions in paired training data.Source: §2 · p. 2

Zero-Shot Learning

  • Zero-shot methods [30] transfer object relationships or map images to semantic word vectors to classify unseen categories, whereas DCC generates full-sentence descriptions.DCC also uses an intermediate-level semantic word-vector representation, but applies transfer to description generation rather than category assignment.Source: §2 · p. 2

Describing New Objects in Context

  • Template-based models describe objects without paired examples but use fixed templates; [21] uses some paired examples, while DCC uses vision data and unpaired text.The cited deep caption model [21] does not address objects with no paired image-sentence data.Source: §2 · p. 2

3. Deep Compositional Captioner

DCC composes descriptions of objects absent from paired image-sentence data by training on unpaired images and text, then paired captions, and transferring knowledge from semantically related words. Its three-stage design combines visual concept recognition, language modeling, and caption generation.

3.1. Deep Lexical Classifier

  • The lexical classifier fine-tunes an ILSVRC-2012-pretrained CNN [28] to predict multiple mined caption concepts and target novel objects using sigmoid cross-entropy.The mined concepts include common adjectives, verbs, and nouns and are not refined, so some may not be strictly visual.Source: §3.1 · p. 3

3.2. Language Model

  • The language model learns sentence structure from unpaired text with an embedding layer, an LSTM, and a next-word prediction layer.At test time, it takes its previous prediction as input, and the model prevents consecutive repetitions of the same word.Source: §3.2 · p. 3

3.3. Caption Model

  • The caption model combines image-concept features and language features to predict words, with visual and language weights learned from paired captions and unpaired text, respectively.Its single-layer linear multimodal unit is designed to make transfer to unseen words straightforward.Source: §3.3 · pp. 3–4
  • On the MSCOCO validation set, DCC achieves a METEOR score of 23.2, compared with 23.7 for [5].The paper reports these results while noting DCC uses a linear multimodal unit.Source: §3.3 · p. 4

3.4. Transferring Information Between Objects

  • Direct transfer copies a related known word’s caption weights to a new word, while delta transfer copies weight changes and preserves pretrained language weights.For visual weights, delta transfer still uses direct transfer; similarity is estimated with word2vec [22] cosine distance, restricting transfer to words in the lexical layer.Source: §3.4 · p. 4

4. Experimental Framework

The framework evaluates DCC on held-out image and video objects, trains lexical and language components from labeled visual data and unpaired text, and assesses whether captions include held-out vocabulary as well as overall sentence quality.

4.1. Datasets

  • The MSCOCO test setup withholds eight objects, while separate experiments examine 642 ImageNet categories and held-out MSVD video objects.For MSCOCO [20], the eight withheld objects are bottle, bus, couch, microwave, pizza, racket, suitcase, and zebra; half the validation set is used for validation and half for testing. The 642 ImageNet categories have no descriptions, and the MSVD held-out objects are zebra, hamster, broccoli, and turtle. Qualitative evaluation also uses 1,952 ILSVRC2015 video snippets featuring objects including whale, fox, hamster, lion, zebra, and turtle.Source: §4.1 · p. 5

4.2. Training the Lexical Classifier

  • The lexical classifier uses VGG-16 features, with video-frame fc7 features mean-pooled across each clip before classification.For paired image-sentence concepts, COCO images labeled with 471 visual concepts train the classifier; the eight held-out objects use MSCOCO or ImageNet images, and new ImageNet concepts use ImageNet images.Source: §4.2 · p. 5

4.3. Training the Language Model

  • The image language model draws on MSCOCO captions, other description corpora, or 60 million external sentences; the video model uses WebCorpus or a caption-text variant.CaptionTxt combines Flickr1M, Flickr30k, Pascal-1k, ImageCLEF-2012, and MSVD descriptions, excluding MSCOCO for image training; the video caption-text variant includes MSCOCO, Flickr-30k, Pascal-1k, and MSVD descriptions.Source: §4.3 · p. 5

4.4. Training the Caption Model

  • Caption-model multimodal weights are trained on paired image-sentence data, using direct or staged delta transfer; the caption model training procedure is shared by image and video.For delta transfer, the language weights are first held fixed while image weights are trained, then both are jointly trained. Table 1 reports F1, BLEU-1, and METEOR comparisons and states that DCC incorporates new words and improves sentence quality.Source: §4.4 · pp. 5–6, Table 1 · p. 6

4.5. Metrics

  • F1 measures whether generated captions include held-out words, while BLEU and METEOR assess overall sentence meaning and fluency.A generated sentence is positive if it mentions a held-out word; a ground-truth image is positive if any annotation mentions that word.Source: §4.5 · p. 6

5. Results

DCC transfers knowledge to describe novel objects in image and video captions, with reported gains in caption metrics and qualitative evidence of contextual composition; the results also identify recurring caption errors.

5.1. Image Description

  • In-domain MSCOCO pretraining performed best, while ImageNet images paired with CaptionTxt or WebCorpus text produced comparable performance.The lexical classifier and language model were pre-trained with unpaired image and text data.Source: Table 3 · p. 6, §5.1 · p. 7
  • Direct transfer improved F1, BLEU, and METEOR more than delta transfer, so it was used for the remaining experiments.DCC without transfer yielded results comparable to LRCN, while transfer improved performance on all reported metrics.Source: §5.1 · pp. 6–7
  • BLEU and METEOR did not decrease for objects present in the held-out training data; with image-sentence examples for all objects, held-out classes scored 69.36 BLEU-1 and 23.98 METEOR.The reported scores are averages for held-out classes.Source: §5.1 · p. 7

5.2. Describing ImageNet Objects

  • DCC accurately described 335 new ImageNet words, including adjectives, and composed captions that placed objects in context.For example, an “otter” was described as sitting in water or standing on a green field depending on the visual context.Source: §5.2 · pp. 7–8, Figure 5 · p. 7
  • DCC errors included omitting the new object, grammatical mistakes, hallucinated objects, and irrelevant descriptions.The paper attributes some errors to poor image recognition or difficulty constructing sentences from visual elements; transferring structurally dissimilar words can also produce grammatical errors.Source: §5.1 · p. 7, §5.2 · p. 8, Figure 6 · p. 8

5.3. Video description

  • F1 increased from 6.0 to 22.2 with ImageNet videos, and both F1 and METEOR increased when outside video data were added.The video evaluation reports average F1 across held-out classes and METEOR across the complete MSVD test dataset.Source: Table 4 · p. 8, §5.3 · p. 8

6. Conclusion

The Deep Compositional Captioner (DCC) describes objects absent from paired caption corpora by combining existing vision datasets with unpaired text and transferring knowledge between semantically related concepts. Its image and video descriptions can incorporate new vocabulary and provide rich descriptions without being limited by paired-data availability.

6. Conclusion

  • DCC describes new objects that are absent from current paired caption corpora.The conclusion presents this as the model’s central capability.Source: §6 · p. 8
  • By combining existing vision datasets with unpaired text and transferring knowledge between related concepts, DCC integrates new vocabulary into image and video descriptions.The method draws on disparate data sources and semantically related concepts.Source: §6 · p. 8
  • DCC provides rich descriptions that are not limited by the availability of paired image-sentence corpora.The conclusion says its quantitative and qualitative results demonstrate this ability.Source: §6 · p. 8

SUPPLEMENTAL MATERIAL · A. Image Description

The supplemental material adds empirical and qualitative results for image and video description, including image-transfer variants, model errors, and examples of descriptions for novel objects.

A. Image Description

SUPPLEMENTAL MATERIAL

A.1. Transferring from Multiple Words · A.2. Transfer with No Language model

Averaging multiple similar-word weights hurts direct transfer but does not significantly affect delta transfer. Without unpaired text training, descriptions can include new words but are often incohesive because the language model has not learned those categories.

A.1. Transferring from Multiple Words

  • Averaging weights across multiple similar words before direct transfer substantially hurts performance.The main-paper approach instead transfers the most similar word’s weights.Source: §A.1 · p. 9
  • Averaging weights before delta transfer does not significantly affect results.This contrasts with averaging before direct transfer.Source: §A.1 · p. 9
  • The multiple-word alternative averages weights across similar words before transferring to new objects in the multimodal unit.The paper’s main approach transfers only the most similar word.Source: §A.1 · p. 9

A.2. Transfer with No Language model

  • Without unpaired text data for language-model training, descriptions of new object categories are poor.The model has not learned good language features for those categories.Source: §A.2 · p. 9
  • Training on paired image-sentence data and unpaired image data can insert new words, but the resulting sentences are not cohesive.The underlying language model has never seen the new object categories.Source: §A.2 · p. 9
  • Without unpaired text training, generated descriptions can be repetitive or ungrammatical.Examples include “A zebra with several zebra and zebra of zebra” and “a pizza bowl of food”.Source: §A.2 · p. 9

A.3. Qualitative Analysis of ImageNet Descriptions · B. Video Description

DCC can describe novel objects in varied visual contexts, but its ImageNet captions also show omissions, misclassifications, hallucinations, and language errors. The merged section also reports the evaluation measures and training setup used for video description.

A.3. Qualitative Analysis of ImageNet Descriptions

  • After transfer, DCC changes subsequent sentence words rather than simply substituting the new object name.For example, it describes a candelabra on a table rather than retaining the pre-transfer vase-and-flowers description; a centrifuge is accurately described even though the pre-transfer sentence omitted the transferred word “refrigerator.”Source: §A.3 · p. 9
  • DCC describes the same object in different visual contexts, such as a gecko held by a person or standing on a tree branch.The examples demonstrate that its descriptions reflect differences in the image context.Source: §A.3 · p. 9
  • DCC sometimes omits the novel object category or misclassifies it, such as describing an alpaca as a sheep.Some descriptions remain relevant despite omitting the new category.Source: §A.3 · p. 9
  • DCC can hallucinate objects that fit learned scene associations or are activated by visual features, especially in single-object images with monochromatic backgrounds.Examples include adding people to an amphitheater scene or describing a man and bench alongside a chainsaw when neither is present; the passage gives these as possible explanations and examples.Source: §A.3 · pp. 9–10
  • Without language-model pretraining on unpaired text, DCC produces less fluent sentences, including repeated words and ungrammatical phrases.The examples include repeating “zebra” and generating “pizza bowl of food.”Source: Figure 8 · p. 10
  • Poor word transfer and weak language features can produce grammatical errors, including repeated phrases or consecutive object names.Examples include “vole bear” and “a unicycle on a unicycle.”Source: §A.3 · p. 10
  • Figure 16 shows examples of captions that are irrelevant to their images.Source: §A.3 · p. 10

B. Video Description

  • Video evaluation reports METEOR across the test dataset and average F1 for four held-out categories.DCC models were trained on videos with four objects removed, and the language models were trained on WebCorpus sentences; the supplied passage gives no metric values.Source: Table 6 · p. 10

B.1. Empirical Results

Training the video-description language model with out-of-domain unpaired text lowered METEOR when ImageNet videos were included, while F1-score increased when they were excluded.

B.1. Empirical Results

  • 29.1 to 28.8: METEOR dropped with out-of-domain unpaired-text training when ImageNet videos were included and transfer was used.This comparison is with Table 4 in the main paper.Source: §B.1 · p. 10
  • 6.0 to 13.3: F1-score increased without ImageNet-video training, but dropped when ImageNet videos were included.Source: §B.1 · p. 10

B.2. Qualitative Results

Qualitative examples show that transfer helps DCC describe novel objects and their contexts more accurately, while also revealing cases of uncertainty and description errors.

B.2. Qualitative Results

  • Image examples show that transfer does not simply substitute a new word for a word present in image-text data.A sentence after transfer can include the new word even when the sentence before transfer did not contain the transferred known word.Source: Figure 9 · p. 11
  • Examples across eight objects show DCC describing objects in different contexts.Figures 10–12 present examples of these descriptions.Source: Figure 10 · p. 12, Figure 11 · p. 13, Figure 12 · p. 14
  • Other examples show misclassified new objects, incorrect context, poor grammar, and descriptions that are incorrect or omit the new category.Poor grammar can result from a poor transfer word or language features learned for the new category; incorrect context is especially common for single objects on monochromatic backgrounds.Source: Figure 13 · p. 15, Figure 14 · p. 15, Figure 15 · p. 15
  • For novel video objects unseen in paired training data, DCC ranks whale and hamster correctly first, but ranks dog above fox while proposing fox second.The lexical classifier is trained with ImageNet and MSVD videos, and the language model with in-domain text.Source: §B.2 · p. 10, Figure 17 · p. 16
  • After transfer, DCC correctly describes hamster, lion, turtle, whale, and zebra, with object and context more accurate than before transfer.For a whale video, the pre-transfer caption describes a woman riding a jet ski, while the post-transfer caption says “A whale is swimming.”Source: §B.2 · p. 10, Figure 18 · p. 16

Related papers

Loading 1511.05284v2…