Source-linked AI summary
Pointing Novel Objects in Image Captioning
Yehao Li, Ting Yao, Yingwei Pan, Hongyang Chao, Tao Mei
TL;DR
Image captioning models trained mainly on image-caption pairs may not describe novel objects in images. The paper introduces LSTM-P, which combines pretrained object learners, pointing-based copying, and object coverage; it reports superior results, including a 60.9% average F1 score on held-out COCO.
Problem
Image-caption pairs contain mainly in-domain objects, limiting captioning models' ability to describe novel objects and scenes in real-world images.
Method
LSTM-P pretrains object learners on recognition data, then balances LSTM word generation with copying recognized objects and trains for sentence-level object coverage.
Results
60.9% average F1 on held-out COCO was reported for LSTM-P, which outperformed the compared state-of-the-art techniques across four general evaluation metrics.
Takeaways & Limitations
Pointing provides a mechanism for integrating recognized novel objects into captions while preserving standard word-by-word generation and encouraging broader object coverage.
Abstract
from arXiv · showhide
Image captioning has received significant attention with remarkable improvements in recent advances. Nevertheless, images in the wild encapsulate rich knowledge and cannot be sufficiently described with models built on image-caption pairs containing only in-domain objects. In this paper, we propose to address the problem by augmenting standard deep captioning architectures with object learners. Specifically, we present Long Short-Term Memory with Pointing (LSTM-P) --- a new architecture that facilitates vocabulary expansion and produces novel objects via pointing mechanism. Technically, object learners are initially pre-trained on available object recognition data. Pointing in LSTM-P then balances the probability between generating a word through LSTM and copying a word from the recognized objects at each time step in decoder stage. Furthermore, our captioning encourages global coverage of objects in the sentence. Extensive experiments are conducted on both held-out COCO image captioning and ImageNet datasets for describing novel objects, and superior results are reported when comparing to state-of-the-art approaches. More remarkably, we obtain an average of 60.9% in F1 score on held-out COCO~dataset.
1. Introduction
Existing captioning models built from image-caption pairs often cover only in-domain objects, limiting description of novel scenes and objects. LSTM-P addresses this through object learners, copying, pointing, and sentence-level object coverage.
- Motivation: Image-caption training pairs provide a shallow view of in-domain objects, hindering generalization to novel scenes or out-of-domain objects.The paper frames novel-object prediction as requiring both vocabulary expansion and integration of recognized objects into captions.
- Approach: LSTM-P augments captioning with object learners and a pointing mechanism that balances LSTM word generation against copying recognized objects.The architecture is introduced to expand vocabulary and control when learned objects are inserted into the output sentence.
- Approach: The model also uses sentence-level coverage of objects to encourage broader object inclusion in generated captions.Coverage is presented as an additional training objective alongside the captioning model.
- Contribution: The work proposes LSTM-P as an architecture for addressing novel-object prediction in image captioning.Its stated contribution is to combine vocabulary expansion with controlled placement of copied novel objects.
2. Related Work
Prior novel-object captioning methods extend RNN-based captioners with additional paired or unpaired data and object detectors or taggers. LSTM-P instead emphasizes dynamically balancing generated words with copied objects and encouraging sentence-level object coverage.
- Novel Object Captioning: Novel-object captioning leverages additional paired or unpaired image, text, and recognition data to describe objects absent from standard captioning data.Existing approaches commonly adapt RNN-based captioning frameworks by injecting novel objects through image taggers or object detectors.
- Summary: LSTM-P augments an RNN language model with object learners pretrained on recognition data, similar to previous approaches [17] [33].Its stated novelty is the dynamic accommodation of language-model generation and copying from learned objects.
- Summary: The pointing mechanism selects when to copy novel objects while balancing copying against standard context-conditioned word generation.Sentence-level coverage is an additional training target intended to encourage global object coverage.
3. Method
LSTM-P augments CNN–LSTM captioning with object learners, copying, and pointing so recognized objects can enter captions while language generation remains coherent. Its training combines sequential modeling with sentence-level object coverage to address visual grounding and reduce object omission or hallucination.
- Architecture: LSTM-P combines CNN–LSTM sentence generation with pre-trained object learners and a copying layer that directly supplies recognized objects.The model integrates visual features at the initial LSTM step, while object-recognition scores feed copying over the object vocabulary.
- Problem formulation: The model represents its vocabulary as the union of paired-caption words and object-recognition words, enabling novel-object insertion through recognition data.Object learners produce distributions over an object vocabulary, which is combined with the caption vocabulary.
- Decoding: At each decoding step, the final word probability is formed by dynamically fusing vocabulary-generation and object-copying distributions.The copying distribution is conditioned on the LSTM output and object-learner predictions, while the pointing weight determines their relative contribution.
- Pointing mechanism: The pointing mechanism uses contextual information to switch between generating a word through LSTM and copying one from recognized objects.Its soft switch is computed from the current input word and LSTM cell output, then fuses the two probability distributions.
- Training objective: LSTM-P jointly optimizes sequential loss for coherent word order and sentence-level coverage loss for addressing image objects.Coverage is treated as an additional training target because standard captioning can lack visual grounding and hallucinate or omit objects.
4. Experiments
Experiments evaluate LSTM-P for novel object captioning on held-out COCO and ImageNet, combining object recognition data with captioning. LSTM-P outperforms compared methods and uses pointing and coverage mechanisms to improve object inclusion and sentence quality.
- 4.1. Dataset and Experimental Settings: The evaluation uses held-out COCO and ImageNet to test novel-object captioning across caption quality, object novelty, F1, and accuracy.Held-out COCO excludes captions containing eight specified objects, while ImageNet evaluates 634 objects absent from COCO.
- 4.3. Performance Comparison: 60.9% averaged F1 on held-out COCO gives LSTM-P a 5.2% relative improvement over the best competitor while outperforming all compared methods across four general metrics.Table 1 reports per-object F1, averaged F1, SPICE, METEOR, and CIDEr for ten models.
- 4.3. Performance Comparison: Pointing improves on direct copying by balancing decoder word generation with copying recognized objects, while sentence-level coverage further improves over LSTM-P−.The coverage objective encourages coherent sentences that address all image objects.
- 4.3. Performance Comparison: 90.06% Novel, 17.67% F1, and 11.91% Accuracy on ImageNet improve over LSTM-C by 24.9%, 7.8%, and 0.7%, respectively.The ImageNet evaluation tests scalability to hundreds of novel objects outside paired image-caption data.
- 4.4. Experimental Analysis: Qualitative examples show LSTM-P copying precise novel objects such as “couch” and covering additional objects such as “blanket” in generated captions.The comparisons include generated sentences, detected objects, and human-annotated ground truth on held-out COCO and ImageNet.
- 4.4. Experimental Analysis: Pointing weights favor copying when object words such as “tennis,” “player,” “ball,” and “racket” are generated, and λ=0.3 achieves the best reported performance.Performance curves over F1average and METEOR are generally ∧-shaped as λ varies from 0 to 1.
5. Conclusions
The paper presents LSTM-P for generating novel objects in image captions by combining object learners, pointing, and sentence-level object coverage. Experiments on held-out COCO and ImageNet validate the approach, including a 60.9% F1 score on held-out COCO.
- 60.9% F1 score is achieved on held-out COCO, establishing the paper’s reported single-model state-of-the-art result.
- LSTM-P addresses novel-object captioning by pre-training object learners on freely available object-recognition data and integrating recognized objects into generated captions.
- Its pointing mechanism balances word generation from the recurrent decoder with copying words from learned objects, while sentence-level coverage encourages more objects to appear.
- Experiments on held-out COCO image captioning and ImageNet validate the model and its analysis.