Deep Compositional Captioning: Describing Novel Object Categories without Paired Training Data

Lisa Anne Hendricks; Subhashini Venugopalan; Marcus Rohrbach; Raymond Mooney; Kate Saenko; Trevor Darrell

Source-linked AI summary

Deep Compositional Captioning: Describing Novel Object Categories without Paired Training Data

Lisa Anne Hendricks, Subhashini Venugopalan, Marcus Rohrbach, Raymond Mooney, Kate Saenko, Trevor Darrell

arXiv:1511.05284v2Nov 17, 2015cs.CVcs.CL

TL;DR

Existing captioning systems rely on paired image-sentence corpora and cannot describe many objects available in recognition datasets. DCC combines unpaired visual and textual data with semantic transfer to compose descriptions of novel objects in images and videos. Its evaluations demonstrate integration of new vocabulary and richer contextual descriptions, while observed errors include omitted objects, grammatical mistakes, and hallucinated objects.

Problem
Existing deep caption models cannot integrate novel objects with known concepts without explicit paired image-sentence examples, despite recognition datasets containing many more object classes.
Method
DCC independently trains a lexical classifier and language model, then transfers knowledge from known to novel objects through a multimodal layer using unpaired image data and external text.
Results
DCC integrates new vocabulary into image and video descriptions and composes contextual descriptions for objects absent from paired corpora.
Takeaways & Limitations
DCC provides descriptions of novel objects and their interactions without requiring paired image-caption data for those objects.
Takeaways & Limitations
DCC sometimes omits new objects, generates grammatically incorrect sentences, or hallucinates objects in its descriptions.

Abstract

from arXiv · show

While recent deep neural network models have achieved promising results on the image captioning task, they rely largely on the availability of corpora with paired image and sentence captions to describe objects in context. In this work, we propose the Deep Compositional Captioner (DCC) to address the task of generating descriptions of novel objects which are not present in paired image-sentence datasets. Our method achieves this by leveraging large object recognition datasets and external text corpora and by transferring knowledge between semantically similar concepts. Current deep caption models can only describe objects contained in paired image-sentence corpora, despite the fact that they are pre-trained with large object recognition datasets, namely ImageNet. In contrast, our model can compose sentences that describe novel objects and their interactions with other objects. We demonstrate our model's ability to describe novel concepts by empirically evaluating its performance on MSCOCO and show qualitative results on ImageNet images of objects for which no paired image-caption data exist. Further, we extend our approach to generate descriptions of objects in video clips. Our results show that DCC has distinct advantages over existing image and video captioning approaches for generating descriptions of new objects in context.

1. Introduction

Existing captioning models depend on paired image-sentence data and cannot compose descriptions for unseen objects. DCC combines unpaired visual and textual resources with transfer between related concepts to describe novel objects in images and videos.

Motivation: Paired image-sentence corpora cover fewer objects than recognition datasets, limiting existing caption models despite their large-scale visual pretraining.ImageNet contains thousands of labeled object classes, whereas caption datasets describe a relatively small variety of objects.
Evaluation: Figure 1 contrasts prior methods’ inability to describe unseen objects such as otters with DCC’s use of independent image and text corpora.DCC composes descriptions without paired image-sentence data for the novel object.
Approach: DCC combines visual groundings from object recognition datasets with unpaired text to integrate novel objects into compositional captions.The model transfers knowledge from semantically related concepts so new visual elements can combine with linguistic expressions learned from paired data.
Approach: DCC uses independently trained lexical and language components, then transfers knowledge to novel objects through a multimodal layer and external text relations.The lexical classifier and language model can also be jointly trained on paired image-sentence data.
Evaluation: The model is evaluated on held-out MSCOCO objects, ImageNet objects absent from caption datasets, and novel objects in YouTube video clips.These evaluations combine empirical results with qualitative demonstrations across image and video description settings.

2. Related Work

Prior deep captioning methods generally combine convolutional visual features with recurrent language generation or multimodal representations. DCC is presented as a substantial modification of these frameworks to compose descriptions for concepts absent from paired data.

Deep Captioning: CNN-RNN captioning systems extract image features with a classification CNN and predict caption words recurrently from visual and linguistic context.Other approaches use multimodal frameworks that combine recurrent language features with visual representations.
Deep Captioning: Visual-concept captioning systems classify objects, attributes, and scenes before conditioning a language model on the detected concepts.DCC most closely resembles a multimodal approach but modifies it to handle concepts never seen in paired image-sentence data.

3. Deep Compositional Captioner

DCC composes captions for objects absent from paired image-sentence data by combining independently learned visual and linguistic representations and transferring knowledge from related known words. Its caption model integrates lexical classification with language structure, supporting descriptions of novel objects in images and videos.

DCC composes novel sentences about objects unseen in paired image-sentence data by using unpaired image data, unpaired text, and semantically related words.The method transfers knowledge between concepts to describe objects present in recognition datasets but absent from caption corpora.
Deep Lexical Classifier: The lexical classifier is a CNN that maps images to semantic concepts and is trained on both caption-related concepts and target objects outside caption datasets.It uses multilabel sigmoid cross-entropy and is initialized from an ILSVRC-2012-pretrained CNN.
Language Model: The language model learns sentence structure from unpaired text by predicting each word from preceding words with an embedding layer, LSTM, and word-prediction layer.Its language features are later combined with image-derived features in the caption model.
Caption Model: The caption model combines image features fI and language features fL through a multimodal unit whose learned weights produce a probability distribution over the next word.Summing visual and linguistic contributions allows the model to form coherent descriptions.
Transferring Information Between Objects: DCC transfers captioning knowledge to unseen words by expanding semantically explicit image features and adapting multimodal weights from related words.Direct transfer copies learned weights, while delta transfer transfers changes learned from paired data without overwriting pretrained language weights.

4. Experimental Framework

The experiments evaluate DCC on images and videos where selected objects lack paired image-sentence training data. They vary visual and textual training sources, transfer strategies, and metrics for new-word integration and sentence quality.

Image description: The held-out MSCOCO training set excludes image-sentence pairs describing eight objects selected from clusters of the 80 annotated MSCOCO objects.Clustering uses word2vec vectors to ensure excluded objects remain similar to included concepts.
Image description: The ImageNet evaluation selects 642 objects absent from MSCOCO but present in the WebCorpus vocabulary, without manual concept pruning.For each class, 75% of images train the lexical classifier and the remainder are used for evaluation; no descriptions exist for these categories.
Video description: The video experiments use 1,970 annotated MSVD clips while holding out paired video-sentence data for some objects during training.Additional qualitative evaluation uses 1,952 ILSVRC2015 detection-video snippets covering 30 objects, including whale, fox, hamster, lion, zebra, and turtle.
Training data: Lexical classifiers use labeled COCO concepts and compare in-domain MSCOCO images with out-of-domain ImageNet images for held-out objects.The experiments therefore vary the source of unpaired visual data used to represent novel categories.
Training data: Language models are trained with MSCOCO captions, CaptionTxt from other image and video datasets, or WebCorpus containing 60 million external sentences.Video language models use WebCorpus or a CaptionTxt variant that includes MSCOCO, Flickr-30k, Pascal-1k, and MSVD descriptions.
Evaluation and transfer: Table 1 compares No Transfer, delta transfer, direct transfer, and LRCN using F1-score for new-word insertion plus BLEU-1 and METEOR for sentence quality.The transfer procedures train multimodal weights; delta transfer first freezes language-model weights before jointly training them with visual weights.

5. Results

DCC integrates novel vocabulary into image and video descriptions by transferring knowledge between related concepts and using unpaired image and text data. Results show improved transfer performance, preserved fluency, and qualitative descriptions of unseen objects, alongside identifiable error modes.

Image Description: DCC integrates new vocabulary into image descriptions cohesively, with direct transfer outperforming delta transfer across F1, BLEU, and METEOR.The direct-transfer method is used for subsequent experiments.
Image Description: 69.36 BLEU-1 and 23.98 METEOR were achieved on held-out classes when training used all image-sentence examples.BLEU and METEOR did not decrease for objects present in the held-out training data.
Image Description: DCC composed descriptions for all eight held-out objects, with semantically similar word pairs such as zebra/giraffe and microwave/refrigerator supporting transfer.Word2vec similarity generally worked well for identifying useful transfer relationships.
Describing ImageNet Objects: DCC accurately described 335 ImageNet words, including nouns, fine-grained categories, and some adjectives, while adapting descriptions to visual context.Transfer could alter more than a noun, producing substantially different contextual sentences.
Describing ImageNet Objects: DCC errors included omitted novel objects, grammatical mistakes, hallucinated objects, and irrelevant descriptions.Examples attribute errors to transfer mismatches, poor image recognition, or difficulty constructing sentences from visual elements.
Video Description: 22.2 F1 versus 6.0 was obtained after adding ImageNet videos, while METEOR also improved with transfer.In-domain text performed slightly better than external text, but external image data benefited video description.

6. Conclusion

The conclusion presents DCC as a captioning model for novel objects absent from paired caption corpora. It combines existing vision datasets, unpaired text, and semantic transfer to generate richer image and video descriptions.

6. Conclusion: DCC describes new objects absent from current caption corpora by integrating vocabulary into generated image and video descriptions.The approach uses existing vision datasets and unpaired text data.

SUPPLEMENTAL MATERIAL

Supplemental experiments add further image and video analyses of DCC, including transfer variations, error examples, external text, and additional novel-object descriptions.

SUPPLEMENTAL MATERIAL: Supplemental material expands image and video analyses with transfer averaging, no-unpaired-text errors, external text, and additional novel-object descriptions.It also includes more DCC descriptions for ImageNet categories and video results.

A.1. Transferring from Multiple Words

DCC can use multiple semantically similar words for transfer, but averaging before direct transfer substantially hurts performance while averaging before delta transfer has little effect.

A.1. Transferring from Multiple Words: Averaging weights across multiple similar words substantially hurts direct-transfer performance.The comparison concerns averaging before transfer.
A.1. Transferring from Multiple Words: Averaging weights before delta transfer does not significantly impact results.This behavior contrasts with direct transfer.
A.2. Transfer with No Language model: Without unpaired text training, generated descriptions are poor because the language model lacks good language features for new object categories.The resulting captions can be repetitive or ungrammatical.

A.3. Qualitative Analysis of ImageNet Descriptions

DCC generates context-sensitive descriptions for novel ImageNet objects, but errors include omitted or misclassified categories, hallucinated context, poor grammar, and irrelevant captions.

A.3. Qualitative Analysis of ImageNet Descriptions: DCC changes subsequent sentence content after transferring a known word to a new object category rather than simply substituting vocabulary.For example, transfer changes descriptions of candelabras and centrifuges even when the original sentence lacks the transferred word.
A.3. Qualitative Analysis of ImageNet Descriptions: DCC describes the same novel object in different visual contexts, as shown by contrasting gecko captions.The examples include a gecko held in a person’s hand and a gecko standing on a tree branch.
A.3. Qualitative Analysis of ImageNet Descriptions: Some captions remain relevant without naming the novel object, or misclassify it while correctly describing other scene elements.An alpaca is one example being classified as a sheep.
A.3. Qualitative Analysis of ImageNet Descriptions: DCC can describe a novel object in an incorrect context and hallucinate likely co-occurring objects, especially in single-object monochromatic images.Examples include people near an amphitheater and a man or bench alongside a chainsaw when those objects are absent.
A.3. Qualitative Analysis of ImageNet Descriptions: Without unpaired text pretraining, captions can repeat words or contain ungrammatical phrases.The examples include repeated “zebra” and “pizza bowl of food.”
A.3. Qualitative Analysis of ImageNet Descriptions: Poor transfer words and weak language features produce grammatical errors, including adjacent object listings and repeated phrases.Examples include “vole bear” and “a unicycle on a unicycle.”
A.3. Qualitative Analysis of ImageNet Descriptions: Some generated captions are irrelevant.This error category is shown in Figure 16.

B.1. Empirical Results

For video description, out-of-domain unpaired text changes performance unevenly: METEOR decreases with ImageNet-video training, while F1-score decreases there but increases without it.

B.1. Empirical Results: METEOR drops from 29.1 to 28.8 when out-of-domain text is used with ImageNet videos during training and transfer.The scores are reported for video description.
B.1. Empirical Results: F1-score drops with ImageNet-video training but rises from 6.0 to 13.3 without ImageNet-video training.These effects are reported when using out-of-domain unpaired text.

B.2. Qualitative Results

Qualitative results show DCC generating multiple likely captions for novel objects in images and videos, with transfer improving object and context descriptions but leaving uncertainty and errors.

B.2. Qualitative Results: After transfer, DCC correctly describes hamster, lion, turtle, whale, and zebra videos while improving object and context accuracy.The whale example changes from describing a woman riding a jet ski to a more accurate description.
B.2. Qualitative Results: DCC is uncertain about some video objects, with its five highest-probability captions naming different animals.This indicates difficulty choosing the correct object.
B.2. Qualitative Results: Image examples show DCC describing different objects in varied contexts, while some outputs omit or misclassify novel categories.Figures 10–13 provide examples of successful and relevant-but-imperfect descriptions.
B.2. Qualitative Results: DCC generates multiple captions for novel video objects, ranked from highest to lowest likelihood.Figure 17 displays the five most likely captions.
B.2. Qualitative Results: Transfer affects sentence structure beyond replacing known words with new object names.The comparison is made for ImageNet video descriptions with and without transfer.