Source-linked AI summary
Decoupled Novel Object Captioner
Yu Wu, Linchao Zhu, Lu Jiang, Yi Yang
TL;DR
Captioners struggle when novel object words are absent from training, while prior remedies depend on additional sentences about those objects. DNOC combines placeholder-based sentence generation with object memory; on held-out MSCOCO, it outperforms state-of-the-art methods for novel-object captioning without extra training sentences.
Problem
Captioners cannot directly generate object words unseen in training, while prior approaches use extra novel-object sentences that may be difficult to collect.
Source: §1 · p. 1, §3.1 · p. 4Method
DNOC generates placeholder-bearing sentences with SM-P, then queries a key-value object memory to replace placeholders with detected object words.
Source: §1 · p. 2, §3.2 · p. 4Results
On held-out MSCOCO, DNOC outperforms state-of-the-art methods for novel-object captioning; its average F1-score is 57.92% versus 55.66% without extra training sentences.
Source: §1 · p. 2, §4.3 · p. 7, §5 · p. 8Takeaways & Limitations
Abstract
from arXiv · showhide
Image captioning is a challenging task where the machine automatically describes an image by sentences or phrases. It often requires a large number of paired image-sentence annotations for training. However, a pre-trained captioning model can hardly be applied to a new domain in which some novel object categories exist, i.e., the objects and their description words are unseen during model training. To correctly caption the novel object, it requires professional human workers to annotate the images by sentences with the novel words. It is labor expensive and thus limits its usage in real-world applications. In this paper, we introduce the zero-shot novel object captioning task where the machine generates descriptions without extra sentences about the novel object. To tackle the challenging problem, we propose a Decoupled Novel Object Captioner (DNOC) framework that can fully decouple the language sequence model from the object descriptions. DNOC has two components. 1) A Sequence Model with the Placeholder (SM-P) generates a sentence containing placeholders. The placeholder represents an unseen novel object. Thus, the sequence model can be decoupled from the novel object descriptions. 2) A key-value object memory built upon the freely available detection model, contains the visual information and the corresponding word for each object. The SM-P will generate a query to retrieve the words from the object memory. The placeholder will then be filled with the correct word, resulting in a caption with novel object descriptions. The experimental results on the held-out MSCOCO dataset demonstrate the ability of DNOC in describing novel concepts in the zero-shot novel object captioning task.
1 INTRODUCTION
Captioning models struggle when novel object words are unseen in training, and collecting sentences about those objects is costly. DNOC generates placeholder-bearing sentences and fills the placeholders using a key-value object memory.
1 INTRODUCTION
- Existing methods either use pretrained language models or extra unpaired sentences containing novel-object words, so those objects are not fully unseen during training.The assumption that such sentences will be available may fail for new products and emerging topics, and they can introduce language biases.Source: §1 · p. 2
- The zero-shot setting requires captioning novel objects without any training sentences about them, relying on a pretrained detector rather than sentence-derived word information.The paper distinguishes this from traditional novel-object captioning, which supplies extra training sentences.Source: §1 · p. 2
- DNOC uses SM-P to generate a sentence with a placeholder, then queries a per-image key-value object memory to fill it with the corresponding object word.This decouples the sequence model from novel-object descriptions.Source: §1 · p. 2
- DNOC outperforms state-of-the-art methods on held-out MSCOCO F1-score without extra training sentences about novel objects.The introduction reports the comparison on F1-score; it does not give the values here.Source: §1 · p. 2
2 RELATED WORK
Prior work spans encoder-decoder captioning and approaches to novel-object description, but many novel-object methods rely on additional data or object-specific mappings.
2.1 Image Captioning
- Recent image-captioning models commonly pair a CNN image encoder with an RNN decoder, with later work adding attention or high-level concept detectors.Earlier approaches used sentence templates or retrieval from sentence pools.Source: §2.1 · p. 2
- Saliency-boosted captioning was proposed to investigate the benefits of low-level cues in language models.Source: §2.1 · p. 3
2.2 Zero-Shot Novel Object Captioning
- Novel-object captioning methods include pretrained taggers and language models, end-to-end systems, forced word inclusion, and copying detections into generated sentences.Lu et al. also generate sentence templates with slots, but require a manually defined category mapping for novel-object words.Source: §2.2 · p. 3
- The paper distinguishes its zero-shot setting by excluding additional sentences and pretrained models for learning novel-object embeddings.The related-work passage states that previous methods use extra novel-object data to train word embeddings.Source: §2.2 · p. 3
3 THE PROPOSED METHOD
DNOC addresses zero-shot novel object captioning by separating sentence generation from object naming: SM-P emits placeholders, and an image-specific detection memory supplies the corresponding labels. Its framework combines an encoder-decoder, memory lookup, and training that teaches placeholder prediction and object retrieval.
3.1 Preliminaries
- The task requires captions containing novel object words absent from both paired image-sentence data and unpaired sentence training data, within an encoder-decoder captioning setup.The setup encodes images into visual features and predicts words sequentially from the image and preceding words.Source: §3.1 · p. 3
3.2 Sequence Model with the Placeholder
- SM-P uses the learned “<PL>” token to represent any novel word, letting the decoder continue generating without taking the unseen word as input.The decoder can use “<PL>” as its next-step input, then the memory supplies the novel description.Source: §3.2 · p. 4
3.3 Key-Value Object Memory
- The object memory stores detected-object appearance features as keys and semantic class labels as values, and reads them by matching a transformed hidden-state query to the keys.At evaluation, the highest-prediction word is used as the query result.Source: §3.3 · pp. 4–5
3.4 Framework Overview
- DNOC generates a sentence with placeholders, builds an image-specific detection-feature/label memory, then uses the preceding hidden state to retrieve labels and replace the placeholders.The framework example produces “a dog is looking at a cake” from “a <PL> is looking at a <PL>”.Source: §3.4 · p. 5
3.5 Training
- Training replaces words shared by the paired-data and detector vocabularies with “<PL>,” teaching SM-P where to emit placeholders and memory which word to return.The final objective simultaneously minimizes the SM-P and object-memory losses.Source: §3.5 · pp. 5–6
4 EXPERIMENTS
Experiments evaluate DNOC on held-out MSCOCO through comparisons with prior methods, component ablations, and qualitative examples.
4.1 The held-out MSCOCO dataset
- For the held-out MSCOCO evaluation, eight objects—bottle, bus, couch, microwave, pizza, racket, suitcase, and zebra—are excluded from training captions and appear in evaluation.The dataset construction excludes paired captions describing at least one of the selected objects.Source: §4.1 · p. 6
4.2 Experimental Settings
- The setup uses Faster R-CNN detections pretrained on all 80 MSCOCO object classes, including the eight held-out objects, and evaluates with F1-score alongside METEOR.The paper notes that METEOR can score captions highly without mentioning novel objects, motivating F1-score as an additional measure.Source: §4.2 · p. 6
4.3 Comparison to the state-of-the-art results
- 57.92% versus 55.66% average F1-score: DNOC exceeds the best prior result without extra sentences, though its METEOR is slightly below LSTM-C with GloVe [41].Table 1 reports per-object and average F1-scores and METEOR for eight novel objects.Source: Table 1 · p. 7, §4.3 · p. 7
- DNOC also achieves higher F1-scores than LRCN on known objects, indicating gains beyond the held-out novel categories.Table 2 compares the methods on six known MSCOCO objects.Source: §4.3 · p. 7, Table 2 · p. 7
4.4 Ablation Studies
- 46.51% in F1-score and 1.08% in METEOR: SM-P without object memory outperforms LRCN, while full DNOC beats that ablation by 11.41% and 1.16%, respectively.The ablation compares the placeholder model, object memory, and detection inputs on held-out MSCOCO.Source: §4.4 · pp. 7–8, Table 3 · p. 7
- F1-score drops sharply with one selected detection and reaches zero with none; performance curves are relatively smooth for two to ten detections.The study varies the number of selected top detection results stored in the object memory.Source: §4.4 · p. 8, Figure 5 · p. 8
4.5 Qualitative Result
- In the zebra example, LRCN says “giraffe,” while DNOC fills SM-P’s placeholder with “zebra” from detection results.Figure 4 presents detected objects, human annotations, LRCN captions, and SM-P templates alongside DNOC’s filled captions.Source: §4.5 · p. 8, Figure 4 · p. 8
5 CONCLUSIONS
DNOC targets novel-object captioning when no sentence containing the novel object is available. Experiments on held-out MSCOCO validate its effectiveness and report that it outperforms state-of-the-art methods for novel-object captioning.
5 CONCLUSIONS
- DNOC addresses novel-object captioning when no sentence about the novel object is available.Source: §5 · p. 8
- The paper proposes a Decoupled Novel Object Captioner framework to generate natural-language descriptions of novel objects.Source: §5 · p. 8
- Experiments on held-out MSCOCO validate DNOC's effectiveness and report that it outperforms state-of-the-art methods for novel-object captioning.Source: §5 · p. 8