PaperlayerView exact revision on arXiv

Source-linked AI summary

Captioning Images with Diverse Objects

Subhashini Venugopalan, Lisa Anne Hendricks, Marcus Rohrbach, Raymond Mooney, Trevor Darrell, Kate Saenko

arXiv:1606.07770v3cs.CVcs.CL

TL;DR

Captioning systems trained on paired image-text data struggle to describe recognizable objects absent from those corpora. NOC jointly trains visual, language, and captioning components on paired data, labeled images, and unannotated text, using distributional embeddings; it describes substantially more novel objects than prior work, while still producing errors such as missed objects and nonsensical or repetitive phrases.

  • Problem

    Paired image-caption supervision limits captioning models’ ability to describe many recognizable objects absent from caption datasets.

  • Method

    NOC jointly trains visual recognition, language modeling, and captioning objectives on paired captions, labeled images, and unannotated text, using distributional embeddings.

  • Results

    NOC achieves 10% higher F1 on unseen COCO objects and 20% higher F1 on ImageNet objects than previous work.

  • Takeaways & Limitations

    NOC describes many more novel objects than prior work while directly extending to ImageNet objects mentioned rarely in image-caption corpora.

  • Takeaways & Limitations

    NOC sometimes fails to describe new objects and can generate incorrect, ungrammatical, nonsensical, or repetitive phrases.

Abstract

from arXiv · show

Recent captioning models are limited in their ability to scale and describe concepts unseen in paired image-text corpora. We propose the Novel Object Captioner (NOC), a deep visual semantic captioning model that can describe a large number of object categories not present in existing image-caption datasets. Our model takes advantage of external sources -- labeled images from object recognition datasets, and semantic knowledge extracted from unannotated text. We propose minimizing a joint objective which can learn from these diverse data sources and leverage distributional semantic embeddings, enabling the model to generalize and describe novel objects outside of image-caption datasets. We demonstrate that our model exploits semantic information to generate captions for hundreds of object categories in the ImageNet object recognition dataset that are not observed in MSCOCO image-caption training data, as well as many categories that are observed very rarely. Both automatic evaluations and human judgements show that our model considerably outperforms prior work in being able to describe many more categories of objects.

1. Introduction

Existing captioning systems rely on paired image-caption data and struggle to describe recognizable objects absent from those datasets. NOC addresses this by jointly learning from multiple data sources and semantic embeddings, improving novel-object description.

  • Motivation: Paired-data captioning systems struggle to generalize to the many recognizable objects absent from image-caption datasets.Scaling paired caption data would be expensive and laborious.
  • Approach: NOC jointly learns captioning, language, and visual recognition objectives from paired captions, unannotated text, and labeled images.The model is trained end-to-end across these sources.
  • Approach: Pre-trained semantic embeddings help NOC describe unseen and rare object categories.The embeddings are incorporated into the captioning model as part of its auxiliary training strategy.
  • Results: NOC substantially outperforms prior work on automated and human evaluations of held-out MSCOCO and ImageNet objects.The experiments include hundreds of ImageNet objects unseen in caption datasets.

2. Related Work

Prior work addresses novel-object captioning through external visual and textual sources, zero-shot learning, or limited novel-concept extensions. NOC follows the broad external-source setting while replacing explicit parameter transfer with jointly trained shared components.

  • Visual Description: Deep visual description models generate captions by conditioning language generation on image features or by embedding images and words in a multimodal space.These approaches generally focus on visual description using image-caption supervision.
  • Novel Object Captioning: Novel-concept captioning has been studied both for small sets of concepts with few paired examples and for many concepts present in external corpora but absent from caption corpora.NOC focuses on the latter setting.
  • Novel Object Captioning: NOC is designed around an architecture whose visual, language, and caption components share parameters while training on different sources.The figure describes simultaneous training with different objectives.
  • Multi-modal and Zero-Shot Learning: Distributional semantics supports related zero-shot tasks by connecting visual and textual information in a joint embedding space.NOC also uses distributional word embeddings for visual description.

3. Novel Object Captioner (NOC)

NOC combines visual recognition, language modeling, and image-caption generation in an end-to-end architecture trained with complementary objectives. Shared parameters, multi-label visual learning, and distributional embeddings preserve and exploit concepts from external image and text data.

  • Architecture: NOC combines a language model using distributional embeddings with a visual recognition model and a caption model trained jointly on multiple data sources.The components share parameters across image-only, text-only, and paired image-caption data.
  • Joint Training: NOC minimizes caption, image-specific, and text-specific losses simultaneously to maintain captioning quality while learning visual concepts and cohesive language.The joint objective is designed to reduce forgetting of objects found only in external data.
  • Image-specific Loss: The image-specific objective trains a multi-label visual classifier so images can receive confidence scores for multiple simultaneous visual concepts.The classifier uses a cross-entropy loss over binary label targets.
  • Caption Model: The caption model predicts each next word from previously generated words and an image by summing language-model and image-model activations.The summed activations are passed through a softmax to obtain the next-word probability.
  • Semantic Embeddings: Distributional embeddings represent words with semantically meaningful vectors, helping the language model relate unseen caption words to semantically similar observed words.GloVe embeddings are used in both input and output transformations.
  • Components: The visual classifier uses VGG-16 with modified final layers for multi-label prediction, while the language model uses an LSTM and softmax output.The visual network’s remaining classification layers are unchanged.

4. Datasets

The experiments combine paired MSCOCO captions with external text and labeled-image resources, then evaluate NOC on held-out MSCOCO objects and ImageNet categories absent from MSCOCO.

  • Data Sources: The dataset section covers the image-description dataset and external text and image datasets used in the experiments.These sources support the model’s multiple training objectives.
  • External Text: Sentences from Gigaword, BNC, UkWaC, and Wikipedia train the LSTM language model.The corpora are tokenized with Stanford CoreNLP.
  • External Text: GloVe embeddings pretrained on 6B external-corpus tokens support a vocabulary refined to 72,700 words.The initial vocabulary contains the 80,000 most frequent combined-corpus tokens before filtering for embeddings.
  • MSCOCO Evaluation: The held-out MSCOCO evaluation removes eight object categories, representing about 10% of the training images and sentences.The categories include bottle, bus, couch, microwave, pizza, racket, suitcase, and zebra.
  • ImageNet Evaluation: A separate evaluation uses approximately 700 ImageNet objects present in the language vocabulary but absent from MSCOCO.The selected words include fine-grained categories, adjectives, and entry-level terms.

5. Experiments on MSCOCO

NOC is evaluated on MSCOCO objects withheld from paired image-caption training, across data-source settings, ablations, and an alternate held-out split. It generally outperforms DCC and benefits from auxiliary objectives, semantic embeddings, and in-domain visual training.

  • Held-out COCO objects: NOC outperforms DCC by 10% F1 on average across eight held-out COCO objects, except couch and microwave.Higher F1 and METEOR indicate more correct recognition of unseen objects and more fluent descriptions.
  • Training data source: In the out-of-domain setting, NOC performs slightly better than DCC on F1 but slightly worse on METEOR across eight held-out objects.NOC can perform inference directly, whereas DCC explicitly transfers parameters from selected seen classes.
  • Training data source: With COCO image training, NOC’s F1 improves considerably even when the language model is trained on the Web Corpus.In the in-domain setting, NOC exceeds DCC by around 10 F1 points and slightly improves METEOR.
  • Ablations: Fixing pre-trained vision weights raises F1 to 39.70, auxiliary objectives raise it to 47.02, and the complete NOC configuration reaches F1 48.79 and METEOR 21.32.The auxiliary objectives tune the visual network on all images, helping retain broad object-recognition ability.
  • Alternate held-out split: NOC remains consistent on a different eight-object COCO split, with F1 and METEOR only somewhat lower than in the original held-out-object experiment.The alternate split contains bed, book, carrot, elephant, spoon, toilet, truck, and umbrella.

6. Experiments: Scaling to ImageNet

NOC is evaluated on novel and rare ImageNet objects absent or infrequent in MSCOCO captions, using automatic and human assessments against DCC. It describes more categories, incorporates novel labels more accurately, and improves rare-word captioning, while still producing semantic, grammatical, and object-identification errors.

  • Novel objects: 638 novel ImageNet categories are evaluated for description coverage, label accuracy, and F1 against DCC.The novel-object set contains ImageNet categories absent from MSCOCO captions.
  • Novel objects: 91.27% (582 of 638) of novel objects are described by NOC, compared with 56.85% (363) for DCC.The metric counts categories whose object labels appear in at least one generated description.
  • Novel objects: 24.74% accuracy and 33.76% F1 are achieved by NOC, versus 11.08% accuracy and 14.47% F1 for DCC.Accuracy measures correct category identification and description across images; F1 uses precision and recall of object mentions.
  • Error analysis: NOC avoids DCC’s need to identify a semantically and syntactically similar known object, reducing one source of captioning error.NOC still commonly fails to name new objects and can generate ungrammatical, nonsensical, or repetitive phrases.
  • Rare objects: 35.15% of rare words are incorporated by NOC, with average F1 47.58%, versus 2.93% and 4.58% for MSCOCO-only training.The rare-word set averages 52 mentions, with a median of 27, in MSCOCO training captions.
  • Human evaluation: Human judgments find NOC and DCC roughly even on shared describable objects, while NOC is better on the union of categories either model can describe.The evaluation separately assesses meaningful word incorporation and image-description quality.

7. Conclusion

The paper presents an end-to-end architecture that combines auxiliary objectives and distributional semantics to caption object classes unseen in paired image-caption data. Across held-out MSCOCO and ImageNet objects, NOC describes more novel categories than prior work while maintaining or improving descriptive quality.

  • Architecture: NOC combines auxiliary training objectives and distributional semantics in an end-to-end architecture for unseen object classes.The approach retains recognition of several hundred object categories while learning caption generation on different images and objects.
  • Results: NOC describes many more novel objects than previous work in both human evaluations and quantitative assessments.The conclusion reports results on held-out MSCOCO objects and several hundred ImageNet objects.
  • Results: 10% higher F1 on unseen COCO objects and 20% higher F1 on ImageNet objects are reported for NOC versus previous work.The paper also reports maintaining or improving descriptive quality.
  • Scope: NOC directly extends to caption ImageNet objects mentioned rarely in image-caption corpora.The conclusion also describes analyses of network modules, training objectives, and data sources.

Supplement

The supplement expands NOC’s qualitative evaluation across varied ImageNet contexts and object categories, and documents comparison and error examples. These examples show contextual descriptions, varied lexical roles, and recurring caption failures.

  • Supplement scope: The supplement includes additional examples of successful descriptions, object diversity, and categorized erroneous descriptions.It also provides quantitative-result details and the Mechanical Turk comparison interface.
  • Contextual descriptions: NOC describes the same novel objects across different backgrounds, actions, and interactions while capturing contextual information.Examples include snowbirds, hyenas, caribou, flounder, and lychees in differing settings.
  • Object diversity: NOC generates descriptions across birds, animals, foods, household objects, clothing, musical instruments, and indoor and outdoor scenes.Some ImageNet nouns are also used appropriately as adjectives, such as chiffon, brownstone, tweed, and woollen.
  • Prior-work comparison: Supplementary comparisons place NOC and DCC captions side by side for images both models can describe.The comparison appears in Figure 9.

A.3. Categorizing Errors

NOC improves coverage of diverse object categories but still produces several error types, ranging from recognition failures to wording and composition problems.

  • NOC sometimes fails to recognize the object or replaces a specific category with a more generic word.Examples include failing to identify a python or describing a wren or warbler as merely a bird.
  • For recognized objects, NOC may repeat words or phrases, misidentify gender, or generate incoherent descriptions.
  • Relevant but Minor Errors: NOC also makes minor errors in counting, age, category identification, repetition, and grammar while often retaining relevant image descriptions.

B. MSCOCO Quantitative Results

The MSCOCO experiments compare NOC with DCC and LRCN on held-out objects, using F1 and METEOR to evaluate generated captions.

  • NOC and DCC are compared on eight held-out objects, with LRCN included as a competitive image-captioning baseline trained on the same split.
  • The evaluation reports average scores across the eight held-out objects and considers objects whose training images come from ImageNet and language data from external text.

B.2. Word-embedding for DCC and NOC

The comparison controls for the different word embeddings used by DCC and NOC. With both models using GloVe, the results indicate that joint training matters more than the embeddings themselves.

  • NOC uses GloVe embeddings in its language model, whereas DCC uses word2vec embeddings to select similar objects for transfer.
  • Replacing DCC’s word2vec embeddings with GloVe does not produce a significant difference in DCC’s results.
  • The controlled comparison indicates that the joint training approach plays a more significant role than the choice of embeddings.

B.3. Joint Training with Auxiliary Objectives

NOC combines image-caption training with auxiliary visual and language objectives, enabling caption generation across varied object categories and contexts while exposing recurring errors and a future adaptation challenge.

  • B.3. Joint Training with Auxiliary Objectives: The overall objective sums image-specific, text-specific, and image-caption losses, with α and β weighting the auxiliary losses.
  • B.3. Joint Training with Auxiliary Objectives: Setting α = 1 and β = 1 gives the best validation performance, while the tested alternatives produce lower F1 and METEOR scores.
  • Future Directions: A proposed future direction is adding new objects from a few image-caption pairs by retraining only on the new data rather than the full dataset.
  • Generated Descriptions: NOC generates descriptions for diverse ImageNet categories, including food, instruments, outdoor scenes, household equipment, vehicles, animals, and clothing.
  • Generated Descriptions: Examples show that NOC often incorporates both the novel object name and the surrounding background context correctly.
  • Generated Descriptions: The model’s recurring errors include repetition, hallucinated objects, counting mistakes, category confusion, and grammatical errors, although some descriptions remain meaningful and relevant.
  • Human Evaluation: Human evaluators compare NOC and prior captions on novel-object incorporation and overall image description using a randomized comparison interface.
Loading 1606.07770v3…