Source-linked AI summary
Incorporating Copying Mechanism in Image Captioning for Learning Novel Objects
Ting Yao, Yingwei Pan, Yehao Li, Tao Mei
TL;DR
Image captioning models trained on image-sentence pairs struggle with novel objects, while obtaining enough paired data is expensive. LSTM-C uses external object-recognition data and combines decoder generation with copying to insert novel objects into captions. Experiments on MSCOCO and ImageNet report improved performance over other novel-object captioning techniques.
Problem
Image captioning models trained on image-sentence pairs have limited ability to describe novel objects, while acquiring large-scale paired data is expensive.
Method
LSTM-C integrates an LSTM decoder with copying from externally trained object detectors, using unpaired visual recognition data in an end-to-end CNN-RNN framework.
Results
LSTM-C reports performance improvements over other novel-object captioning techniques on MSCOCO and ImageNet, including relative gains of 4.3% in Novel, 4.9% in F1, and 17.8% in Accuracy over NOC on ImageNet.
Takeaways & Limitations
Combining generative sentence modeling with copying enables the framework to describe novel objects using external visual recognition knowledge.
Takeaways & Limitations
The approach has difficulty detecting and copying visually similar novel objects such as “couch” and “microwave.”
Abstract
from arXiv · showhide
Image captioning often requires a large set of training image-sentence pairs. In practice, however, acquiring sufficient training pairs is always expensive, making the recent captioning models limited in their ability to describe objects outside of training corpora (i.e., novel objects). In this paper, we present Long Short-Term Memory with Copying Mechanism (LSTM-C) --- a new architecture that incorporates copying into the Convolutional Neural Networks (CNN) plus Recurrent Neural Networks (RNN) image captioning framework, for describing novel objects in captions. Specifically, freely available object recognition datasets are leveraged to develop classifiers for novel objects. Our LSTM-C then nicely integrates the standard word-by-word sentence generation by a decoder RNN with copying mechanism which may instead select words from novel objects at proper places in the output sentence. Extensive experiments are conducted on both MSCOCO image captioning and ImageNet datasets, demonstrating the ability of our proposed LSTM-C architecture to describe novel objects. Furthermore, superior results are reported when compared to state-of-the-art deep models.
1. Introduction
Image captioning uses CNN-RNN encoder-decoder models, but learning directly from image-sentence pairs limits description of novel objects and requires expensive annotation. LSTM-C addresses this by combining sentence generation with copying from externally trained object detectors.
- Image captioning generates complete natural sentences from images, with potential applications in robotic vision and assistance for visually impaired people.
- CNN-RNN sequence models encode image content with a CNN and generate sentences with a decoder RNN.
- Training directly on image-sentence pairs limits these models’ ability to describe objects outside the training data, such as “suitcase.”Large-scale caption annotation is also intellectually expensive and time-consuming.
- LSTM-C incorporates external visual recognition datasets and a copying mechanism into the CNN-RNN captioning framework.The copying mechanism selects novel objects and places them at appropriate positions in generated sentences.
- The framework combines standard word-by-word LSTM generation with copying detected objects for novel-object captioning.The proposal specifically targets integrating generative and copying mechanisms in sentence generation.
2. Related Work
Image captioning research spans template-based, search-based, and language-based approaches, while novel-object captioning adds external paired or unpaired data. LSTM-C focuses on unpaired recognition data and combines decoder generation with end-to-end copying of novel objects.
- Image captioning methods are grouped into template-based, search-based, and language-based approaches.
- Template-based methods align sentence fragments with visual content, whereas search-based methods select semantically similar sentences from a sentence pool.
- Language-based models learn a joint distribution of visual content and text to generate sentences with more flexible syntactical structures.
- Novel Object Captioning: Novel-object captioning uses additional paired image-sentence data or unpaired image/text data to describe objects absent from standard captioning data.
- Novel Object Captioning: LSTM-C focuses on freely available unpaired object-recognition data and integrates decoder generation with copying in an end-to-end trainable framework.
3. Image Captioning with Copying Mechanism
LSTM-C extends CNN–RNN image captioning with a copying mechanism that uses externally trained object detectors to describe novel objects. It combines standard word generation and object copying during decoding in an end-to-end trainable framework.
- Approach: LSTM-C incorporates copying into image-caption decoding to describe novel objects from external recognition data.The framework uses object classifiers trained on unpaired recognition datasets alongside paired image-sentence training.
- Optimization: Training minimizes the negative log probability of target sentences while regularizing image, textual, copying, generative, and LSTM parameters.The objective also incorporates text-specific loss on external sentence data to maintain novel-object handling.
- Approach: CNN image features initialize the LSTM, which then generates the caption sequentially from previous word inputs and hidden states.The image representation is injected at the initial time, followed by word embeddings during decoding.
- Copying mechanism: The copying mechanism estimates probabilities for candidate objects detected in the image and can place them directly into the output sentence.Copying extends rote memorization from sequence learning to selecting appropriate detected objects during caption generation.
- LSTM with Copying Mechanism: For words shared by the generative and copying vocabularies, LSTM-C combines both probabilities; words exclusive to either vocabulary use the corresponding mechanism.The final distribution uses a tradeoff parameter λ and a softmax normalization term.
4. Experiments
Experiments on held-out MSCOCO and ImageNet evaluate LSTM-C against regular and novel-object captioning methods. Results show that combining generative sentence modeling with copying improves novel-object description, while visually confusable objects remain difficult.
- Evaluation on held-out MSCOCO: LSTM-C outperformed LRCN, DCC, and NOC across averaged F1 and METEOR on held-out MSCOCO.All five novel-object captioning models also outperformed LRCN on description quality and novelty.
- Evaluation on held-out MSCOCO: LSTM-C achieved the best F1 performance for six of the eight novel objects on held-out MSCOCO.The exceptions were couch and microwave, which were difficult to detect precisely because of visual similarity to other objects.
- Evaluation on ImageNet: On ImageNet, LSTM-C (One hot+Glove) improved over NOC (One hot+Glove) by 4.3% in Novel, 4.9% in F1, and 17.8% in Accuracy.The experiment used MSCOCO for the CNN-plus-RNN training system while scaling to hundreds of novel ImageNet objects.
- Qualitative analysis: Qualitative examples show LSTM-C predicting more precise novel objects than LRCN, such as bus instead of hydrant.The examples compare detected objects and generated sentences on held-out MSCOCO and ImageNet.
5. Discussions and Conclusions
The paper presents LSTM-C, an end-to-end image-captioning framework that combines LSTM sentence generation with copying from externally detected objects. Experiments on MSCOCO and ImageNet validate the proposal, while larger object sources and video remain future directions.
- LSTM-C combines standard word-by-word LSTM generation with copying from detected objects to predict novel objects in image captions.The framework leverages external visual recognition and integrates the generative and copying mechanisms end to end.
- Experiments on MSCOCO image captioning and ImageNet validate the proposed framework and analysis.
- Learning more objects from large-scale image benchmarks and applying the proposal to video are identified as future work.The paper specifically mentions YFCC-100M as a possible additional image source.