PaperlayerView exact revision on arXiv

Source-linked AI summary

nocaps: novel object captioning at scale

Harsh Agrawal, Karan Desai, Yufei Wang, Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, Stefan Lee, Peter Anderson

arXiv:1812.08658v3cs.CVcs.AIcs.CLcs.LG

TL;DR

Image captioners generalize poorly beyond the limited concepts in their training data, while earlier novel-object evaluation covered only eight classes. This paper introduces nocaps, a large-scale benchmark covering 394 novel classes, and finds its final models outperform direct COCO transfer but remain below human performance.

  • Problem

    Image captioning models trained on limited concept vocabularies generalize poorly to real-world images, while prior novel-object evaluation covered only eight selected classes.

    Source: §1 · pp. 1–2

  • Method

    The paper introduces nocaps, a benchmark of OpenImages images with human captions, using COCO image-caption pairs plus OpenImages labels and bounding boxes for training.

    Source: §1 · p. 2, §6 · p. 8

  • Results

    Final models significantly outperform direct COCO transfer but remain well below the human baseline.

    Source: §6 · p. 8

  • Takeaways & Limitations

    Nocaps provides a large-scale test of novel object captioning, covering 394 classes rare or absent in training captions versus eight in the earlier proof-of-concept dataset.

    Source: §6 · p. 8

    Open questions remain about combining stronger object detectors and language models and choosing between learning and inference techniques.

    Source: §5 · p. 7

Abstract

from arXiv · show

Image captioning models have achieved impressive results on datasets containing limited visual concepts and large amounts of paired image-caption training data. However, if these models are to ever function in the wild, a much larger variety of visual concepts must be learned, ideally from less supervision. To encourage the development of image captioning models that can learn visual concepts from alternative data sources, such as object detection datasets, we present the first large-scale benchmark for this task. Dubbed 'nocaps', for novel object captioning at scale, our benchmark consists of 166,100 human-generated captions describing 15,100 images from the OpenImages validation and test sets. The associated training data consists of COCO image-caption pairs, plus OpenImages image-level labels and object bounding boxes. Since OpenImages contains many more classes than COCO, nearly 400 object classes seen in test images have no or very few associated training captions (hence, nocaps). We extend existing novel object captioning models to establish strong baselines for this benchmark and provide analysis to guide future work on this task.

1. Introduction

nocaps introduces a large-scale benchmark for captioning images with novel objects, testing whether models can combine caption data with object-detection data to describe visual concepts beyond those covered by caption corpora. It also analyzes existing methods and reports their performance relative to a COCO-only baseline and humans.

1. Introduction

  • COCO-trained captioning models generalize poorly to real-world visual concepts beyond their limited training coverage, constraining uses such as vision assistance and image retrieval.The introduction gives accordions and dolphins as examples of objects COCO-trained models typically cannot describe.Source: §1 · p. 1
  • Earlier novel-object captioning was evaluated on a proof-of-concept dataset with only eight held-out classes chosen for similarity to COCO, leaving large-scale performance uncertain.The paper argues that caption models should leverage detection datasets to describe objects absent from caption corpora.Source: §1 · pp. 1–2
  • nocaps is the first large-scale novel-object captioning benchmark, covering nearly 400 novel object classes with Open Images validation and test images.Its validation and test sets contain 4,500 and 10,600 images, respectively, each annotated with 11 human-generated captions.Source: §1 · p. 2
  • The benchmark uses COCO image-caption pairs and Open Images detection annotations, letting models learn caption generation and many more visual concepts from separate data sources.Training includes no additional paired image-caption data: COCO provides 118K images and 80 object classes, while Open Images provides 1.7M images with boxes for 600 classes and labels for 20K categories.Source: §1 · p. 2
  • On nocaps, two existing approaches [2] [27] outperform a COCO-only baseline yet remain far below humans on CIDEr and SPICE across in-, near-, and out-of-domain subsets.The subsets reflect how similar depicted objects are to COCO classes.Source: §1 · p. 2
  • The paper investigates two existing state-of-the-art models and proposes improvements and simple heuristics that significantly improve constrained beam search on nocaps.It contrasts the models with human performance.Source: §1 · p. 2

2. Related Work

Related work spans transfer-based, detector-explicit, and architecture-agnostic approaches to novel object captioning, while nocaps positions itself between limited-concept human-annotated datasets and broad but noisier web-harvested datasets.

Novel Object Captioning

  • Transfer-based methods treat novel object captioning as domain adaptation from COCO to Open Images, separating visual and textual components and leveraging detection data and external text [8] [15] [42].The Deep Compositional Captioner and Novel Object Captioner train the components with separate losses and jointly with available image-caption data.Source: §2 · p. 2
  • Neural Baby Talk and Decoupled Novel Object Captioner generate slotted templates filled with detector concepts, while LSTM-C adds a copying mechanism for detector-predicted words [27] [45] [49].These methods use object-detector outputs explicitly in caption generation.Source: §2 · p. 2
  • Architecture-agnostic methods use Constrained Beam Search to enforce selected words, while PS3 iteratively estimates complete captions as training targets using an expectation-maximization-inspired algorithm [2, 3, 9].PS3 uses Constrained Beam Search as a subroutine for images containing novel objects.Source: §2 · p. 2
  • This work evaluates Neural Baby Talk and Constrained Beam Search on nocaps; both had recently claimed state of the art on a proof-of-concept novel-object-captioning dataset [27] [2] [15].The passage identifies them as the two approaches investigated on the more challenging benchmark.Source: §2 · pp. 2–3

Image Caption Datasets

  • Directly annotated datasets offer controlled captions and multiple references but limited object classes, whereas web-filtered datasets offer diverse concepts but risk non-visual text and weak human-evaluation baselines [1] [6] [28] [41].The passage notes that filtered datasets may lack enough captions per image for automatic metrics to correlate well with human judgments.Source: §2 · p. 3
  • nocaps aims to bridge these dataset paradigms with 10 human reference captions per image and more visual concepts than COCO, and is described as the only benchmark where humans outperform state-of-the-art models on automatic evaluation.This is the authors’ claim to the best of their knowledge.Source: §2 · p. 3

3. nocaps

nocaps uses complex Open Images images and human captions to benchmark captioning of visually novel concepts, with dataset analysis and evaluation guidelines that distinguish its visual and linguistic diversity from COCO.

3.1. Caption Collection

  • 4,500 validation images and 10,600 test images were selected to favor complex scenes and balanced object-class representation, averaging 4.0 classes and 8.0 instances per image.The selection excluded images with unknown or nonzero rotation and images containing only one object category; it included all images with more than 6 unique classes.Source: §3.1 · p. 3
  • 166,100 captions were collected from crowd-workers, with 11 captions per image: one sampled for human performance and 10 used as evaluation references.Workers were primed with object-class keywords because pilot studies found this produced more accurate and descriptive captions; they were told they need not mention every keyword.Source: §3.1 · p. 4, Figure 3 · p. 4

3.2. Dataset Analysis

  • 600 object classes span nocaps versus 80 in COCO; nocaps images average 4.0 classes and 8.0 instances, versus 2.9 classes and 7.4 instances in COCO.Nocaps has no single-class images, compared with 20% in COCO; nearly 22% of nocaps images have more than 6 classes, versus less than 10% in COCO. Its captions average 11 words versus 10 in COCO and use a larger vocabulary and more unique 2-, 3-, and 4-grams.Source: §3.2 · p. 4

3.3. Evaluation

  • Submissions must not use additional human-generated paired image-caption data or ground-truth object annotations; external text, knowledge bases, and detection datasets are allowed.The benchmark uses an EvalAI server, maintains a separate nocaps (XD) leaderboard for unrestricted extra data, and focuses on CIDEr and SPICE alongside Bleu, Meteor, and ROUGE.Source: §3.3 · p. 5
  • The benchmark divides images into in-domain, near-domain, and out-of-domain subsets according to whether their objects belong to COCO-related classes.The 119 in-domain classes comprise 80 COCO classes and 39 additional Open Images classes frequently mentioned in COCO captions; out-of-domain images contain none of these classes. Test subsets contain 1,311 in-domain, 7,406 near-domain, and 1,883 out-of-domain images, with out-of-domain expected to be most challenging for COCO-only models.Source: §3.3 · pp. 5–6

4. Experiments

The experiments establish nocaps baselines by adapting Neural Baby Talk (NBT) and applying Constrained Beam Search (CBS) to NBT and UpDown, with Open Images detections used to expose models to novel object concepts.

Experiments

  • NBT uses region-linked text slots and Open Images detections, while CBS is also applied to the UpDown captioner as a detection-based baseline.The Open Images Faster R-CNN detector used by both approaches achieves 54% detection mAP@0.5; NBT’s language model is trained with Visual Genome image features and fixed GloVe embeddings.Source: §4 · p. 6

Constrained Beam Search (CBS)

  • CBS uses a 24-state finite-state machine for up to three detected-object constraints and selects the highest-log-probability caption satisfying at least two.Constraints exclude 39 Open Images classes considered parts or rarely mentioned, suppress overlapping higher-order classes at IoU 0.85 or above, and use the top three detections by confidence.Source: §4 · p. 6

Language Embeddings

  • CBS incorporates forward ELMo representations and a fixed ELMo-initialized decoder softmax to improve generalization to unseen or rare words.Because captions are decoded left-to-right, the method uses only ELMo’s forward representation rather than its full bidirectional model.Source: §4 · p. 6

Training and Implementation Details

  • All models are trained on COCO, tuned on the nocaps validation set, and optimized with cross-entropy rather than reinforcement-learning fine-tuning.CBS constraints are selected using the described filtering heuristic rather than randomly, which preliminary nocaps experiments found improved performance.Source: §4 · p. 6

5. Results and Analysis

UpDown + ELMo + CBS improves on the COCO-trained baseline on nocaps, but remains far below human performance, especially on out-of-domain images. The analyses identify tradeoffs across COCO performance, constraint filtering, language models, and object detections.

Overall results

  • UpDown + ELMo + CBS beats the COCO-trained UpDown baseline by 19 CIDEr, but trails humans by 12 CIDEr overall and by 25 CIDEr on out-of-domain instances.The authors conclude that captioning in the wild remains an open challenge.Source: §5 · p. 6, Table 2 · p. 7

Do models optimized for nocaps maintain their performance on COCO?

  • Large nocaps gains correspond to COCO losses of 20 CIDEr and 3 SPICE, suggesting that transfer between the datasets remains a challenge.The authors suggest limiting this degradation as a direction for future work.Source: §5 · p. 7, Table 3 · p. 7

How important is constraint filtering?

  • Without the 39-class blacklist and overlap filtering, UpDown + ELMo + CBS loses 8 CIDEr and 3 SPICE on nocaps validation, with most losses from blacklisted classes.Constraint-based decoding improves UpDown and NBT, particularly on out-of-domain captions, but depends heavily on constraint quality.Source: §5 · p. 7, Table 3 · p. 7

Qualitative analysis

  • On near-domain and out-of-domain images, models confuse novel objects such as rifles and dolphins with familiar objects, and difficult examples expose repetition and incoherent descriptions.Even models using Open Images data leave substantial room for improvement in grounding and grammar; the authors identify stronger detectors and language models, and how to combine learning with inference, as open directions.Source: §5 · pp. 7–8, Figure 6 · p. 8

Do better language models help in CBS?

  • ELMo improves out-of-domain performance over learned-only embeddings, while in-domain and near-domain results are similar; rare COCO mentions can teach language but not visual grounding.In the learned-only setting, words absent from COCO receive random initialization.Source: §5 · pp. 7–8, Table 3 · p. 7

Do better object detectors help?

  • Ground-truth detections improve UpDown by 9 CIDEr and 0.6 SPICE across splits compared with predicted detections, indicating that better detectors could raise nocaps performance.The ground-truth detections are sorted by decreasing area and undergo the same constraint filtering as predicted detections.Source: Table 3 · p. 7, §5 · p. 8

6. Conclusion

The paper introduces nocaps as a larger, more rigorous benchmark for novel object captioning and establishes baselines that improve over direct COCO transfer but remain below human performance.

6. Conclusion

  • Nocaps provides 166,100 human-generated captions for 15,100 images with more than 500 object classes, including 394 rare or absent training-caption classes versus 8 in the prior dataset [14].The benchmark also includes twice as many evaluation captions per image to improve automatic evaluation fidelity.Source: §6 · p. 8
  • The final models improve significantly over direct transfer from COCO but perform well below the human baseline.The authors identify substantial room for improvement on novel object captioning.Source: §6 · p. 8
  • Analysis finds benefits from pretrained word embeddings and language models, stronger object detectors, and heuristics for selecting detections to mention.The authors note that object detectors can also be a source of further improvements.Source: §6 · pp. 8–9

Appendix

The appendix documents nocaps data collection, benchmark details, baseline implementation, and examples from the validation split.

Section 1

  • Section 1 describes the data-collection interface for nocaps.Source: Appendix · p. 11

Section 2

  • Section 2 provides qualitative examples from the nocaps validation split.Source: Appendix · p. 11

Section 3

  • Section 3 provides additional details about the nocaps benchmark.Source: Appendix · p. 11

Section 4

  • Section 4 gives implementation details for the baseline models.Source: Appendix · p. 11

Section 5

  • Section 5 shows predicted captions on the in-domain, near-domain, and out-of-domain nocaps validation subsets.Source: Appendix · p. 11

1. Data Collection Interface

The caption-collection interface uses Amazon Mechanical Turk and displays a subset of image object categories as keywords; workers are explicitly told they need not mention them.

1. Data Collection Interface

  • The AMT interface presents a subset of object categories in the image as keywords for caption gathering.Source: Figure 7 · p. 11
  • Workers are told that mentioning displayed keywords is optional, and the other instructions are similar to the interface in [7].Source: Figure 7 · p. 11

2. Example Reference Captions from nocaps in-domain near-domain out-of-domain

Figures 8 and 9 show nocaps validation examples from the in-domain, near-domain, and out-of-domain subsets, each paired with 10 human-written reference captions.

3. Additional Details about nocaps Benchmark

The section defines nocaps evaluation subsets by COCO-caption exposure and examines how visual and linguistic similarity relate to dataset differences. COCO-trained models’ performance trends suggest nocaps captions remain linguistically similar to COCO when visual similarity is controlled.

3.1. Evaluation Subsets

  • The 119 in-domain classes comprise 80 manually mapped COCO classes and 39 additional Open Images classes mentioned over 1,000 times in COCO captions; all remaining classes are out-of-domain.Examples of the additional classes include table, plate, and tree.Source: §3.1 · p. 14
  • In-domain classes occur much more frequently in COCO captions, while out-of-domain classes are relatively infrequent rather than necessarily absent, making them harder to learn from COCO.Figure 10 plots class mentions on a log scale, with in-domain classes in orange and out-of-domain classes in blue.Source: §3.1 · p. 14, Figure 10 · p. 14
  • The 87 ignored Open Images classes include parts, broad super-categories, solo categories without co-occurring classes, and rare categories absent from the validation and test splits.Examples include vehicle registration plate and wheel; sports equipment and home appliance; chime and stapler; and armadillo and pizza cutter, respectively.Source: §3.1 · p. 14

3.2. T-SNE Visualization in Visual Feature Embedding Space

  • The visual-feature t-SNE shows in-domain classes near COCO, while out-of-domain classes include both visually distant clusters and concepts filling gaps among sparse COCO-related clusters.The plot uses reduced Faster-RCNN image features for object classes.Source: Figure 11 · p. 15

3.3. T-SNE Visualization in Linguistic Feature Embedding Space

  • The linguistic-feature t-SNE shows out-of-domain classes that are linguistically distant from in-domain classes, alongside linguistically similar fine-grained classes absent from the in-domain split.The plot uses reduced GloVe vectors for object classes.Source: Figure 12 · p. 16

3.4. Linguistic Similarity to COCO

  • UpDown exceeds human performance on the most COCO-visually-similar nocaps decile, then declines as images become less similar; the best model degrades less, suggesting visual rather than linguistic differences.The analysis uses average cosine distance in FC7 CNN feature space to the 10 closest COCO images as a proxy for visual similarity.Source: §3.4 · p. 17, Table 4 · p. 17

4. Additional Implementation Details for Baseline Models

The section describes modifications to the original Neural Baby Talk implementation so it can caption images containing novel objects in nocaps.

4.1. Neural Baby Talk (NBT)

  • The authors modify the original Neural Baby Talk implementation to produce captions for nocaps images containing novel objects.Source: §4.1 · p. 18

Grounding Regions for Visual Words … 5. Example Model Predictions

The section details adaptations to NBT and constrained beam search for nocaps, including region grounding, visual and language features, and decoding constraints. It also describes ELMo implementation choices and illustrates predictions on challenging images.

Grounding Regions for Visual Words; Bottom-Up Visual Features

  • An Open Images V4 Faster-RCNN detector supplies NBT region proposals and detects 601 classes, including nocaps novel classes; bottom-up features from a Visual Genome detector feed attention and pointer inputs.COCO proposals are filtered at IoU and confidence thresholds of 0.5, with up to 18 per image. The model uses 2048-dimensional region features, top-36 class-agnostic features, and their mean-pooled representation for language-model input.Source: Grounding Regions for Visual Words · p. 18, Bottom-Up Visual Features · p. 18

Fine-grained Class Mapping

  • NBT uses a separate MLP to predict fine-grained labels for detected object classes, extending the COCO mapping with Open Images classes and removing overlapping labels from the fine-grained set.For example, “man” and “woman” are retained as Open Images object classes rather than fine-grained classes of “person.”Source: Fine-grained Class Mapping · p. 18

Visual Word Prediction Criterion

  • NBT ties a region to a caption slot when its class confidence exceeds 0.5 and it overlaps ground truth by IoU greater than 0.5, but drops the ground-truth-class match criterion to avoid suppressing fine-grained detections.The relaxation may introduce false-positive detections in captions, but the passage says it prevents reduced visual grounding.Source: Visual Word Prediction Criterion · p. 18

Finite State Machine

  • The constrained beam search finite-state machine uses 24 states to produce captions satisfying at least two of three constraints, with extra states for multiword constraint phrases.The base FSM uses eight states for three single-word constraints; each two-word phrase requires four additional states and each three-word phrase eight.Source: Finite State Machine · p. 19, Figure 13 · p. 19, Figure 15 · p. 20

Determining Constraints

  • Filtering out 39 rare, broad, or part classes significantly improves constrained beam search, while suppressing overlapping objects can also slightly improve performance across UpDown, UpDown + GD, and UpDown + ELMo.In these experiments, detected objects are ranked by confidence and the top three are used as constraints.Source: Determining Constraints · pp. 18–19, Table 6 · p. 20

Integrating UpDown Model with ELMo

  • UpDown + ELMo combines ELMo representations with trainable scalar weights and initializes a fixed softmax layer from word embeddings to support predictions for unseen objects.An added fully connected layer with tanh aligns the softmax-layer and LSTM hidden-state dimensions.Source: Integrating UpDown Model with ELMo · p. 21

Other details of using ELMo

  • The ELMo model vocabulary combines COCO caption words and singular and plural Open Images labels, limited to ELMo’s output vocabulary; the model is optimized with SGD and selected on nocaps val.The experiments use the official ELMo TensorFlow checkpoint trained on the 1 Billion Word Language Model Benchmark.Source: Other details of using ELMo · p. 21

5. Example Model Predictions

  • Figures 16 and 17 show captions from existing approaches for challenging nocaps images, marking CBS constraints in blue and NBT visual words in red.Source: Figure 16 · p. 22, Figure 17 · p. 23
Loading 1812.08658v3…