Source-linked AI summary
Boosting Image Captioning with Attributes
Ting Yao, Yingwei Pan, Yehao Li, Zhaofan Qiu, Tao Mei
TL;DR
Image captioning requires natural language generation that captures both image content and relationships, while existing CNN-plus-RNN methods do not explicitly use high-level semantic attributes. The paper introduces end-to-end LSTM-A architectures that combine image representations and attributes in varied ways, achieving improved COCO performance and first-place leaderboard ranking. The authors identify learning more attributes and enlarging the generated vocabulary as future work.
Problem
Image captioning must generate natural sentences capturing image objects, scenes, and their relationships, while many existing models rely primarily on image representations rather than explicit high-level attributes.
Method
The paper introduces end-to-end Long Short-Term Memory with Attributes architectures that feed image representations and detected high-level attributes into LSTMs in different placements and time patterns.
Results
The proposed LSTM-A achieves improved COCO captioning performance and ranks first on the COCO captioning Leaderboard.
Takeaways & Limitations
Jointly exploiting image representations and attributes, together with architectural exploration of their relationship, supports stronger image captioning performance on COCO.
Takeaways & Limitations
Future work includes learning more attributes from large-scale benchmarks, analyzing attribute quantity, and enlarging the generated vocabulary.
Abstract
from arXiv · showhide
Automatically describing an image with a natural language has been an emerging challenge in both fields of computer vision and natural language processing. In this paper, we present Long Short-Term Memory with Attributes (LSTM-A) - a novel architecture that integrates attributes into the successful Convolutional Neural Networks (CNNs) plus Recurrent Neural Networks (RNNs) image captioning framework, by training them in an end-to-end manner. To incorporate attributes, we construct variants of architectures by feeding image representations and attributes into RNNs in different ways to explore the mutual but also fuzzy relationship between them. Extensive experiments are conducted on COCO image captioning dataset and our framework achieves superior results when compared to state-of-the-art deep models. Most remarkably, we obtain METEOR/CIDEr-D of 25.2%/98.6% on testing data of widely used and publicly available splits in (Karpathy & Fei-Fei, 2015) when extracting image representations by GoogleNet and achieve to date top-1 performance on COCO captioning Leaderboard.
1 INTRODUCTION
Image captioning seeks to generate complete, natural sentences from images, but must capture both visual content and relationships among objects or scenes. This work asks how high-level attributes can complement image representations in CNN-plus-RNN captioning architectures.
- Image captioning aims to describe image content with complete, natural sentences for applications including robotic vision and assistance for visually impaired people.
- Generating captions is challenging because models must represent objects or scenes and express their relationships in natural language.
- Recent CNN-plus-RNN methods adapt sequence-to-sequence learning, encoding image representations before generating captions one word at a time.
- These methods do not explicitly incorporate high-level semantic information, motivating attributes as complementary knowledge to image representations.
- The study varies where and when image representations and attributes enter the RNN, including attributes-only, different input orders, and repeated inputs across time steps.
2 RELATED WORK
Image captioning research includes template-based, search-based, and language-based approaches. This paper belongs to the language-based group and studies jointly using image representations and attributes while exploring how their architectural relationship should be modeled.
- Image captioning methods are organized into template-based, search-based, and language-based approaches.
- Template-based methods use predefined grammatical structures and align sentence fragments with detected image content.
- Search-based methods select or copy semantically similar human-generated sentences, limiting scalability and preventing novel descriptions.
- Language-based models learn a joint visual-textual probability distribution, mainly using neural networks to generate flexible, novel sentences.
- This work jointly exploits image representations and high-level attributes while exploring their mutual relationship through architectural design.
- Unlike semantic attention to locally previous words, the paper uses attributes holistically as complementary representations.
3 BOOSTING IMAGE CAPTIONING WITH ATTRIBUTES
The paper formulates image captioning as an end-to-end CNN-plus-RNN sequence-generation problem that incorporates detected high-level attributes alongside image representations. It explores five LSTM-A variants differing in where and when these inputs enter the LSTM.
- Problem formulation: Attribute detectors produce an image-level attribute probability vector using weakly supervised Multiple Instance Learning.The vector A represents the probability distribution over high-level attributes for the image.
- Problem formulation: The captioning objective minimizes the negative log probability of the correct sentence conditioned on image representations and detected attributes.The sentence probability is decomposed across sequential words, preserving contextual relationships during generation.
- Framework: LSTM-A integrates detected high-level attributes into LSTM-based CNN-plus-RNN image captioning architectures.The models encode image representations and/or attributes before decoding a target sentence.
- Architecture variants: Five variants explore whether attributes or image representations are used alone, encoded first, or injected repeatedly during decoding.LSTM-A1 uses only attributes; LSTM-A2 and LSTM-A3 vary encoding order; LSTM-A4 and LSTM-A5 vary repeated inputs during decoding.
4 EXPERIMENTS
Experiments evaluate LSTM-A variants and competing captioning systems on COCO using standard captioning metrics. The results show strong performance for attribute-integrated models, with LSTM-A3 and LSTM-A5 differing across metrics and LSTM-A3 leading the online leaderboard comparison.
- Dataset: COCO experiments use 82,783 training images, 5,000 validation images, and 5,000 testing images, each with five human-annotated descriptions.The official test annotations are unavailable, so the study follows widely used prior-work splits.
- Settings: The study extracts GoogleNet image features and 1,000-way MIL-predicted attribute probability vectors, then evaluates BLEU@N, METEOR, ROUGE-L, and CIDEr-D.Rare words occurring fewer than five times are discarded, producing an 8,791-word vocabulary.
- Compared approaches: The comparison includes non-attention, attention-based, publicly available, and five proposed LSTM-A variants on COCO.LSTM-A1 through LSTM-A5 represent different ways of incorporating image representations and attributes.
- Performance comparison: 98.6% CIDEr-D is reported for LSTM-A on COCO with GoogleNet image representations, described as the highest reported performance using those representations.Across seven evaluation metrics, LSTM-A is reported to outperform the listed state-of-the-art techniques.
- Performance comparison: LSTM-A3 performs best among the proposed variants on BLEU@1 and METEOR, while LSTM-A5 performs best on the other five evaluation metrics.LSTM-A2 improves over LSTM-A1 but remains below LSTM-A3; feeding attributes at each time step improves over LSTM-A3.
- Online testing: LSTM-A3 ranks first on the official COCO leaderboard across all evaluation metrics on both c5 and c40 testing sets.Using ResNet-152 for attribute-detector training further raises CIDEr-D to 104.9% on c5 and 105.3% on c40.
5 DISCUSSIONS AND CONCLUSIONS
LSTM-A integrates image representations with high-level attributes for image captioning and evaluates architectural variants for combining them. Experiments on COCO show improvements over other captioning techniques, while attribute coverage and vocabulary expansion remain future directions.
- LSTM-A explores both image representations and high-level attributes for image captioning.The architectures augment attributes to complement image representations during sentence generation.
- The study varies where and when image representations and attributes enter the recurrent architecture.The variants modify the placement and moment of feeding the two representation types.
- Performance improvements are clearly observed compared with other captioning techniques, with LSTM-A ranking first on the COCO captioning Leaderboard at the time.
- Future work includes learning more attributes, analyzing the impact of attribute count, and enlarging the generated sentences' word vocabulary.The proposed extensions include using large-scale image benchmarks such as YFCC-100M.