Source-linked AI summary
Guided Open Vocabulary Image Captioning with Constrained Beam Search
Peter Anderson, Basura Fernando, Mark Johnson, Stephen Gould
TL;DR
Existing captioning models generalize poorly to out-of-domain images with novel scenes or objects. The paper uses constrained beam search and pretrained embeddings to incorporate image tags at test time without retraining, achieving state-of-the-art out-of-domain results and improving in-domain and ImageNet captioning.
Problem
Image captioning models trained on datasets covering limited objects and interactions do not generalize well to out-of-domain images containing novel scenes or objects.
Method
The method uses constrained beam search to force selected image-tag words or phrases into outputs and fixed pretrained embeddings to expand the vocabulary to unseen tag words.
Results
The approach achieves state-of-the-art out-of-domain captioning on held-out MSCOCO, improves in-domain performance, and significantly improves ImageNet captions using ground-truth labels.
Takeaways & Limitations
Image tags can be incorporated during decoding rather than training, allowing existing captioning models to use test-time text fragments without retraining when the source changes.
Takeaways & Limitations
Object-mention F1 should not be expected to increase substantially because the underlying image tagger remains the same.
Abstract
from arXiv · showhide
Existing image captioning models do not generalize well to out-of-domain images containing novel scenes or objects. This limitation severely hinders the use of these models in real world applications dealing with images in the wild. We address this problem using a flexible approach that enables existing deep captioning architectures to take advantage of image taggers at test time, without re-training. Our method uses constrained beam search to force the inclusion of selected tag words in the output, and fixed, pretrained word embeddings to facilitate vocabulary expansion to previously unseen tag words. Using this approach we achieve state of the art results for out-of-domain captioning on MSCOCO (and improved results for in-domain captioning). Perhaps surprisingly, our results significantly outperform approaches that incorporate the same tag predictions into the learning algorithm. We also show that we can significantly improve the quality of generated ImageNet captions by leveraging ground-truth labels.
1 Introduction
The paper targets poor generalization of captioning models to novel scenes and objects by incorporating scalable image-tag information during decoding. Constrained beam search provides guidance, while pretrained embeddings support previously unseen tag words.
- Motivation: Captioning models trained on datasets covering relatively few objects and interactions generalize poorly to out-of-domain images with novel scenes or objects.This limitation hinders real-world use on images in the wild.
- Motivation: Existing image collections often include semantic attributes or object annotations, and image taggers can generate such fragments for new concepts.These annotations are easier to scale than captioning data.
- Method: The approach uses constrained beam search to guarantee selected words or phrases appear in RNN outputs while leaving syntax and additional details unconstrained.The search enforces constraints expressible with a finite-state machine.
- Method: Fixed pretrained input and output word embeddings let the RNN generalize from similar words when its vocabulary is expanded with unseen tag words.This addresses text fragments containing words absent from the original RNN vocabulary.
- Evaluation: The method uses predicted image tags as decoding constraints to achieve state-of-the-art out-of-domain captioning while improving the base model on in-domain data.The evaluation uses a held-out version of MSCOCO.
2 Related Work
Prior work combines CNN image features, RNN language models, tags, detectors, or pretrained embeddings, typically integrating semantic attributes during learning. This paper instead incorporates text fragments during decoding, enabling looser coupling and avoiding caption-model retraining when the text source changes.
- Neural captioning: Neural captioning commonly combines a pretrained CNN image encoder with an RNN decoder that predicts each next word from prior words and the image.Decoding typically searches output sequences with beam search.
- Out-of-domain captioning: The Deep Compositional Captioner combines separately pretrained CNN tagger and RNN language models, then fine-tunes them jointly on image-caption data.It targets out-of-domain captioning with objects lacking caption training data.
- Out-of-domain captioning: The Novel Object Captioner also uses pretrained word embeddings in both language-model input and output layers.It is described as contemporary work to this paper.
- Related approaches: Other approaches use specialized celebrity or landmark detectors, while several works incorporate semantic attributes into caption-model training for in-domain data.These approaches differ from decoding-time text-fragment integration.
- Distinction: Unlike prior approaches, this work inserts text fragments during decoding rather than learning, allowing test-time information and avoiding retraining when the fragment source changes.The paper presents this as a loosely coupled approach to out-of-domain captioning.
3 Approach
The approach combines constrained beam search with an existing CNN-RNN captioning model to enforce sequence constraints while preserving the model’s ability to choose syntax and details.
- Constrained Beam Search: Beam search approximately decodes RNN outputs by retaining the b most likely partial sequences at each timestep.The candidate set contains all possible next-word extensions before pruning to beam size b.
- Constrained Beam Search: Constraints are represented by a finite-state machine whose states correspond to separate search beams.State transitions route each sequence extension to the beam associated with the resulting FSM state.
- Constrained Beam Search: Accepting-state beams contain sequences that satisfy all constraints recognized by the FSM during left-to-right decoding.In the example, Beam 3 contains captions satisfying both constraint sets.
- Constraint Types: Conjunctive disjunctive constraints require one vocabulary word from each set, allowing the model to choose among alternative word forms.WordNet maps image tags to vocabulary words sharing relevant lemmas; the algorithm maintains one beam for each subset of disjunctive constraints.
- Constraint Types: Subsequence constraints require a phrase to appear in the generated caption, with FSM states increasing linearly with phrase length.The number of states equals the number of phrase words plus one.
- Captioning Model: The evaluated base model is an LRCN with a CNN feature extractor and two 1,000-unit LSTM layers receiving language and image features.The bottom layer receives the embedded previous word, while the top layer receives its output concatenated with static CNN features.
4 Experiments
The experiments evaluate constrained decoding with predicted image tags on out-of-domain and in-domain MSCOCO data, then test ground-truth synset constraints for ImageNet captioning. The approach improves caption metrics and human judgments, while performance depends on tagger quality and tag influence.
- The MSCOCO dataset contains 123,293 images, including 82,783 training images and 40,504 validation images, each with five human-annotated captions.
- Out-of-Domain Image Captioning: The base model receives an out-of-domain object-mention F1 score of 0 because excluded objects do not appear in its caption training data.
- Out-of-Domain Image Captioning: Predicted-tag constrained beam search significantly outperforms prior work on SPICE, METEOR, and CIDEr across out-of-domain and in-domain test data.The evaluation uses one to four predicted tags as constraints and compares against DCC and NOC.
- Out-of-Domain Image Captioning: Adding more tag constraints shifts captions further from the base model: 36% are identical with two tags, compared with 3% with four tags.The authors describe this as trading off influence between image tags and the captioning model.
- Out-of-Domain Image Captioning: The approach improves unseen-object F1 with four tags, but a large increase is not expected because it reuses the same underlying image tagger.
- Captioning ImageNet: Ground-truth synset constraints improve ImageNet caption structure, although the task remains extremely challenging because many classes are poorly represented in caption-training data.
- Captioning ImageNet: Base+Syn captions were judged equally good or better than human captions in 22% of pairwise evaluations, versus 11% for the base model.The 22% comprises 12% judged better and 10% equally good; 78% were judged equally poor or worse than human captions.
- Captioning ImageNet: ImageNet performance is strongest in super-categories with some caption-training representation, including birds, mammals, and dogs.
5 Conclusion and Future Research
The paper investigates constrained beam search for incorporating image tag predictions into image captioning and reports state-of-the-art out-of-domain results. It also identifies more powerful taggers and constrained search within EM learning as future directions.
- Constrained beam search enforces finite-state constraints on generated output sequences while enabling image tag predictions to guide captioning.The approach is applied to out-of-domain captioning on held-out MSCOCO.
- The approach achieves state-of-the-art results for out-of-domain image captioning on held-out MSCOCO using image tag predictions.
- Future work will investigate more powerful image taggers and constrained beam search within an expectation-maximization algorithm for weakly supervised caption learning.