nocaps: novel object captioning at scale

Harsh Agrawal; Karan Desai; Yufei Wang; Xinlei Chen; Rishabh Jain; Mark Johnson; Dhruv Batra; Devi Parikh; Stefan Lee; Peter Anderson

Source-linked AI summary

nocaps: novel object captioning at scale

Harsh Agrawal, Karan Desai, Yufei Wang, Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, Stefan Lee, Peter Anderson

arXiv:1812.08658v3Dec 20, 2018cs.CVcs.AIcs.CLcs.LG

TL;DR

Image captioning models trained on limited visual concepts generalize poorly to the long-tailed objects encountered in the wild. The paper introduces nocaps, a large-scale benchmark using COCO captions alongside Open Images detection data, and shows that extended novel-object captioning models improve over direct COCO transfer but remain below humans.

Problem
Captioning models trained on datasets covering a tiny fraction of real-world visual concepts generalize poorly to images containing objects absent from their caption corpora.
Method
The paper introduces nocaps, combining COCO image-caption pairs with Open Images object labels and bounding boxes, and evaluates extended novel object captioning models.
Results
166,100 captions describe 15,100 nocaps images, while extended models improve significantly over direct COCO transfer but remain well below the human baseline.
Takeaways & Limitations
nocaps provides a rigorous large-scale benchmark and analysis showing benefits from pretrained language resources, better object detectors, and heuristics for selecting object detections.
Takeaways & Limitations
Open questions remain about how best to combine stronger object detectors and language models, and how much should be learned versus handled by inference methods such as CBS.

Abstract

from arXiv · show

Image captioning models have achieved impressive results on datasets containing limited visual concepts and large amounts of paired image-caption training data. However, if these models are to ever function in the wild, a much larger variety of visual concepts must be learned, ideally from less supervision. To encourage the development of image captioning models that can learn visual concepts from alternative data sources, such as object detection datasets, we present the first large-scale benchmark for this task. Dubbed 'nocaps', for novel object captioning at scale, our benchmark consists of 166,100 human-generated captions describing 15,100 images from the OpenImages validation and test sets. The associated training data consists of COCO image-caption pairs, plus OpenImages image-level labels and object bounding boxes. Since OpenImages contains many more classes than COCO, nearly 400 object classes seen in test images have no or very few associated training captions (hence, nocaps). We extend existing novel object captioning models to establish strong baselines for this benchmark and provide analysis to guide future work on this task.

1. Introduction

Image captioning models trained on limited caption datasets generalize poorly to the long-tailed visual concepts found in the wild. nocaps introduces a large-scale benchmark that tests whether models can use object detection data to describe novel objects.

Motivation: Captioning models trained on COCO Captions typically describe common objects but not concepts such as accordions or dolphins.Their poor in-the-wild generalization limits applications including assistance for people with impaired vision and natural-language image retrieval.
Evaluation: The benchmark evaluates in-domain, near-domain, and out-of-domain images according to whether they contain COCO classes, mixed classes, or only novel classes.These subsets measure performance across increasing differences between the caption and detection data domains.
Task: Novel object captioning uses alternative data sources, such as object detection datasets, to describe objects absent from caption corpora.Novel objects have detection annotations but are not present in the caption training data.
Benchmark: 394 rare or absent novel object classes appear in nocaps, compared with 8 in the earlier proof-of-concept dataset.This represents a fifty-fold increase in novel object classes and leaves the large-scale performance of previous methods open to question.
Benchmark: nocaps contains 15,100 images and 166,100 human-generated captions, with 4,500 validation and 10,600 test images annotated with 11 captions each.Training combines 118K COCO captioned images with 1.7M Open Images detection images, without additional paired image-caption training data.
Baselines: Two existing approaches improve over a COCO-only baseline but remain well below human performance on nocaps.The study investigates Neural Baby Talk and Constrained Beam Search, including CBS applied to NBT and UpDown.

2. Related Work

Prior novel object captioning work used specialized architectures, detector-driven templates, or architecture-agnostic decoding and self-training methods. nocaps broadens evaluation beyond the small, similarity-biased proof-of-concept setting with substantially greater visual and caption diversity.

Novel Object Captioning: Deep Compositional Captioner and Novel Object Captioner transfer visual concepts across source and target distributions.Their setting treats COCO as the source distribution and Open Images as the target distribution.
Detector-Based Approaches: Neural Baby Talk and Decoupled Novel Object Captioner generate slotted templates filled with concepts identified by object detectors.LSTM-C similarly augments recurrent sentence decoding with detector-related information.
Architecture-Agnostic Approaches: Constrained Beam Search enforces selected detector-predicted words in captions during inference, while PS3 turns completed captions into iterative training targets.PS3 uses CBS as a subroutine in an expectation-maximization-inspired algorithm.
Evaluation Gap: nocaps evaluates two contemporary approaches, Neural Baby Talk and Constrained Beam Search, on a substantially larger benchmark than the earlier proof-of-concept dataset.The methods had previously claimed state-of-the-art performance on that smaller dataset.
Dataset Diversity: nocaps images average 4.0 object classes and 8.0 instances per image, versus 2.9 classes and 7.4 instances in COCO Captions.Captions are also longer on average: 11 words versus 10.
Dataset Design: nocaps provides 10 reference captions per image and more visual concepts than COCO, aiming to fill a gap between existing image-caption datasets.The authors report that humans outperform state-of-the-art models in automatic evaluation on this benchmark.

3. nocaps

nocaps is a large-scale benchmark designed to test captioning models on visually novel concepts using Open Images images and limited paired-caption supervision. Its collection and evaluation procedures emphasize diverse, complex images, accurate human captions, and controlled use of external data.

Caption collection: Priming annotators with ground-truth object categories improves caption accuracy and descriptiveness for rare, fine-grained classes such as red panda.Without priming, red panda was incorrectly described as a brown rodent; priming produced a correct description.
Caption collection: 15,100 Open Images validation and test images receive 11 human-generated captions each, yielding 166,100 captions.One caption supports a human baseline, while 10 serve as automatic-evaluation references.
Dataset analysis: Compared with COCO, nocaps spans 600 object classes, averages 4.0 versus 2.9 classes per image, and 8.0 versus 7.4 instances per image.The selected images emphasize multiple objects and rare co-occurrences rather than single-category iconic views.
Dataset analysis: Nocaps captions average 11 words versus COCO’s 10 and use more diverse vocabularies and n-gram compositions despite object-category priming.Table 1 compares equally sized samples of 4,500 images and 22,500 captions from each dataset.
Evaluation: Evaluation uses CIDEr and SPICE, separates in-domain, near-domain, and out-of-domain subsets, and restricts paired-caption training to COCO Captions 2017.External text corpora, knowledge bases, and object-detection datasets may be used, but ground-truth object annotations are prohibited in submissions.

4. Experiments

The experiments extend contemporary novel-object captioning approaches to establish initial nocaps baselines. They evaluate NBT and CBS-based systems, including CBS applied to both NBT and UpDown.

Experimental methods: The study evaluates Neural Baby Talk and Constrained Beam Search, applying CBS to both NBT and the UpDown captioner.These systems provide an initial measure of state-of-the-art performance on nocaps.
Baseline models: UpDown uses visual features from a Faster R-CNN detector trained on Visual Genome to provide a strong paired-caption-only baseline.NBT instead generates region-linked templates and fills slots with detector-associated visual concepts.
Experimental methods: CBS decodes captions with a 24-state finite-state machine, selecting the highest-log-probability caption satisfying at least two object constraints.Constraints can include up to three selected objects and two- or three-word phrases.

5. Results and Analysis

Novel object captioning methods improve substantially over COCO-only baselines, but remain far below human performance, especially for out-of-domain images. Analysis attributes gains to constraint filtering, pretrained language models, and better object detections, while exposing trade-offs and remaining grounding and grammar problems.

≈19 CIDEr separates the best model from the COCO-trained UpDown baseline, but humans remain ≈12 CIDEr ahead.The largest remaining gap occurs on out-of-domain instances, at ≈25 CIDEr.
Significant nocaps gains correspond to losses of ≈20 CIDEr and ≈3 SPICE on COCO.The authors identify limiting cross-dataset degradation as a future-work focus.
Constraint filtering greatly improves UpDown and NBT, particularly out of domain, but performance depends heavily on constraint quality.Removing the 39-class blacklist and overlap filtering reduces UpDown + ELMo + CBS validation performance by ≈8 CIDEr and ≈3 SPICE.
ELMo helps most on out-of-domain subsets, while COCO-trained embeddings perform comparably in-domain and near-domain.Rare occurrences of nocaps object names in COCO can support linguistic modeling but not visual grounding.
Ground-truth object detections produce large gains across all splits, including 9 CIDEr and 0.6 SPICE for UpDown.This suggests detector improvements may yield commensurate benchmark gains.
Qualitative examples show COCO-only models confusing novel objects with visually similar known objects, while stronger models still need better grounding and grammar.Examples include rifle confused with baseball bat and dolphin confused with bird.

6. Conclusion

nocaps is a substantially larger benchmark for novel object captioning, and the authors extend existing methods to establish baselines and analyze sources of improvement. The resulting models improve over direct COCO transfer but remain well below human performance, leaving substantial room for progress.

166,100 human-generated captions describe 15,100 images containing more than 500 unique object classes.The benchmark includes 394 rare-or-absent novel object classes versus 8 in the earlier proof-of-concept dataset.
The benchmark increases rare-or-absent novel object classes fifty-fold, from 8 to 394, and doubles evaluation captions per image.
Extended novel object captioning models improve significantly over direct COCO transfer but remain well below the human baseline.
Analysis finds benefits from pretrained language resources, better object detectors, and heuristics selecting which detections to mention.

Appendix

The appendix outlines the paper’s organization, covering data collection, qualitative examples, benchmark details, model implementation, and validation captions across three image subsets.

The paper describes its data-collection interface, qualitative validation examples, benchmark details, baseline implementation, and predicted captions.The predicted captions cover in-domain, near-domain, and out-of-domain validation subsets.

1. Data Collection Interface

The caption-collection interface primes AMT workers with image object categories as keywords while explicitly making keyword use optional.

AMT workers see a subset of image object categories as keywords, but instructions state that mentioning them is optional.

2. Example Reference Captions from nocaps in-domain near-domain out-of-domain

The nocaps validation examples span in-domain, near-domain, and out-of-domain images, with ten human reference captions illustrating varied descriptions of salient content.

Reference captions: Ten reference captions accompany each image, capturing varied salient content for evaluation.The examples include descriptions of people, animals, objects, settings, and activities.
Dataset subsets: In-domain, near-domain, and out-of-domain subsets contain images with differing object-class relationships to COCO.Orange categories indicate in-domain classes, while blue categories indicate out-of-domain classes.
Caption content: Reference captions may omit ground-truth object classes while describing other salient image content.This reflects the captioning instructions used for the benchmark examples.
Example content: Examples include captions describing dogs and people, cameras, lanterns, food trucks, hot tubs, breweries, and related scenes.Multiple captions describe the same image from different perspectives and with varying specificity.
Example content: The examples also show fine-grained variation in wording for shared scenes, such as lanterns, cameras, food trucks, and brewing equipment.Descriptions range from object-focused references to broader scene summaries.
Example content: Captions cover both individual objects and broader environments, including patios, rooms, restaurants, streets, and parking lots.This broad coverage demonstrates the range of content represented in the reference captions.

3. Additional Details about nocaps Benchmark

The benchmark defines domain subsets by relating Open Images classes to COCO and examining their caption frequency and visual or linguistic similarity. COCO-trained captioners perform best on visually similar nocaps images, while performance declines as visual distance increases.

Domain subset construction: 80 COCO classes are manually mapped to Open Images classes to identify in-domain categories.An additional 39 non-COCO Open Images classes are included when mentioned more than 1,000 times in COCO training captions.
Domain subset construction: Out-of-domain classes occur less frequently than in-domain classes in COCO Captions, but are not necessarily absent.Their relative infrequency makes these concepts harder to learn from caption training data.
Excluded categories: 87 Open Images classes are excluded from image subset selection because they are parts, broad super-categories, or unsuitable solo categories.Examples include vehicle registration plates, sports equipment, and other categories that overlap with parent or existing classes.
Caption analysis: The collection methodology follows COCO but adds keyword priming, which may introduce linguistic differences between nocaps and COCO.The analysis evaluates COCO-trained models while controlling for visual similarity to COCO.
Caption analysis: UpDown exceeds human performance on the most COCO-similar nocaps decile, then drops consistently as images become visually more distinct.This pattern supports high linguistic similarity between COCO and nocaps after controlling for visual variation.

4. Additional Implementation Details for Baseline Models

The implementation details extend Neural Baby Talk so it can caption images containing novel objects in nocaps.

Neural Baby Talk: NBT is modified to produce captions for images containing novel objects present in nocaps.The modifications target the original authors’ implementation of Neural Baby Talk.

Grounding Regions for Visual Words

The baseline systems ground caption words in detected regions and incorporate object constraints during decoding, while ELMo supplies external word representations and supports unseen-object vocabulary handling.

Grounding regions for visual words: NBT uses detector region proposals and caption templates with slots tied to specific image regions.An Open Images-trained Faster-RCNN detector provides proposals for nocaps object classes.
Grounding regions for visual words: Up to 18 COCO region proposals per image are extracted with IoU and confidence thresholds of 0.5.The proposal pipeline reduces highly overlapping regions and low-confidence detections.
Grounding regions for visual words: NBT combines attention over region features, spatial CNN features, and FC7 features concatenated with word embeddings.The region-level features support visual words, while the spatial and FC7 features provide holistic image context.
Grounding regions for visual words: A separate MLP predicts fine-grained object classes because detector labels can be coarser than caption terminology.For example, captions may distinguish cheesecake, cupcake, and coffeecake despite a coarse detector label such as cake.
Grounding regions for visual words: The grounding criteria require confidence above 0.5 and IoU above 0.5, but the ground-truth class-match criterion is removed for nocaps.Removing it preserves fine-grained detections such as man and woman when the ground-truth label is person, while potentially allowing false positives.
Constrained beam search: CBS forces captions to include detected object words, using filtered top-3 constraints and selecting captions satisfying at least two constraints.The FSM supports up to three selected objects, including two- and three-word phrases.
Constrained beam search: Filtering blacklisted classes and overlapping objects improves constrained beam search performance, with both strategies performing best for the ELMo model.The ablations compare removing classes, removing overlap, and removing both heuristics.
Integrating UpDown Model with ELMo: ELMo representations combine character embeddings with hidden outputs from two LSTM layers using trainable scalar weights.The ELMo parameters are fixed except for the combination weights when used as an external representation.