Source-linked AI summary
Image Captioning with Deep Bidirectional LSTMs
Cheng Wang, Haojin Yang, Christian Bartz, Christoph Meinel
TL;DR
Image captioning requires learning long-range visual-language interactions while translating images into sensible sentences. The paper develops end-to-end deep bidirectional LSTM architectures with CNN image encoding, evaluates them on captioning and retrieval benchmarks, and reports competitive generation and stronger retrieval performance. The study also examines internal states and uses data augmentation to support deep-model training.
Problem
Image captioning requires modeling visual objects, semantic interactions, and sentence-level translation, while unidirectional models lack future textual context for reasoning about preceding words.
Method
The paper combines CNN image encoding with separate bidirectional and deep LSTM architectures that jointly learn high-level visual-language embeddings end to end.
Results
Bidirectional LSTM models achieve highly competitive caption-generation performance and significantly outperform recent methods on image-sentence retrieval across Flickr8K, Flickr30K, and MSCOCO.
Takeaways & Limitations
Bidirectional history-and-future context supports competitive captioning and strong retrieval without requiring additional mechanisms such as object detection or attention models.
Takeaways & Limitations
Direct comparison is constrained by differing MSCOCO dataset divisions and visual models across related methods.
Abstract
from arXiv · showhide
This work presents an end-to-end trainable deep bidirectional LSTM (Long-Short Term Memory) model for image captioning. Our model builds on a deep convolutional neural network (CNN) and two separate LSTM networks. It is capable of learning long term visual-language interactions by making use of history and future context information at high level semantic space. Two novel deep bidirectional variant models, in which we increase the depth of nonlinearity transition in different way, are proposed to learn hierarchical visual-language embeddings. Data augmentation techniques such as multi-crop, multi-scale and vertical mirror are proposed to prevent overfitting in training deep models. We visualize the evolution of bidirectional LSTM internal states over time and qualitatively analyze how our models "translate" image to sentence. Our proposed models are evaluated on caption generation and image-sentence retrieval tasks with three benchmark datasets: Flickr8K, Flickr30K and MSCOCO datasets. We demonstrate that bidirectional LSTM models achieve highly competitive performance to the state-of-the-art results on caption generation even without integrating additional mechanism (e.g. object detection, attention model etc.) and significantly outperform recent methods on retrieval task.
1. INTRODUCTION
Image captioning requires modeling visual objects, semantic interactions, and the translation from visual understanding to sensible sentences. This work proposes deep bidirectional LSTM architectures that use history and future context to learn high-level visual-language embeddings and achieve competitive captioning and retrieval results.
- Image captioning must recognize visual objects, model semantic interactions, and translate visual understanding into sensible sentence descriptions.
- The proposed end-to-end model encodes sentences with bidirectional LSTMs, images with CNNs, and jointly learns a visual-language space.
- Bidirectional LSTMs exploit both past and future textual context, producing complementary captions whose combination can yield more sensible descriptions.
- Deeper LSTM variants increase nonlinear transition depth to learn higher-level visual-language embeddings while avoiding dramatic parameter growth.
- The work visualizes bidirectional hidden-state evolution to analyze how visual context conditions word generation over time.
- Across Flickr8K, Flickr30K, and MSCOCO, bidirectional models are highly competitive with state-of-the-art caption generation and significantly outperform recent retrieval methods.
2. RELATED WORK
Image captioning is part of multimodal representation learning, where shared concepts help bridge the semantic gap between modalities. Prior approaches include template-based, neural-network-based, and attention-based methods.
- Multimodal representation learning uses shared concepts across modalities to bridge the semantic gap, with image captioning as one instance.
- Template-based captioning generates sentence structures using detected objects, attributes, phrases, and relationships within images.
- Neural captioning methods use CNNs and recurrent networks, including multimodal language models and encoder-decoder pipelines with LSTM sentence encoding.
- Attention-based approaches incorporate a human-visual-system-inspired mechanism that can visualize attended content and improve caption generation.
3. MODEL
The model combines CNN image encoding with bidirectional and deeper LSTM architectures to learn visual-language representations and generate captions using both past and future context. It is trained end-to-end with joint forward/backward losses and uses augmentation to reduce overfitting.
- Long Short Term Memory: The LSTM maintains memory through input, forget, and output gates that regulate writing, retention, and exposure of the cell state.Its recurrent state update uses current input, previous hidden state, and previous memory state.
- Bidirectional LSTM: Bidirectional processing uses separate forward and backward LSTMs so predictions can exploit both preceding and succeeding sentence context.The forward model processes from t = 1 and the backward model from t = T.
- Bidirectional LSTM: The multimodal model combines a CNN, Text-LSTM, Multimodal LSTM, and Softmax layers to encode images and sentences in a shared semantic space.The Multimodal LSTM captures visual-word correlations while decoding the next-word distribution.
- Deeper LSTM architecture: Two deeper variants increase nonlinear transition depth: Bi-S-LSTM stacks LSTM layers, whereas Bi-F-LSTM inserts a fully connected transition layer with shortcut connections.The fully connected transition is intended to increase depth without causing parameter size to grow dramatically.
- Overfitting prevention: Multi-crop, multi-scale, and vertical-mirror augmentation increase image variation, with the combined procedure producing roughly 40 times more image-sentence training pairs.The augmentations are applied online through random transformations during training.
- Training and decoding: Training minimizes a joint loss formed by accumulating Softmax losses from forward and backward directions using SGD and BPTT.The resulting model predicts captions from both directions and selects the final sentence using summed word probabilities.
4. EXPERIMENTS
Experiments analyze how bidirectional LSTM representations evolve during caption generation and evaluate captioning, retrieval, efficiency, and comparison robustness across three benchmark datasets.
- Experimental Setup: Experiments use Flickr8K, Flickr30K, and MSCOCO to assess qualitative behavior, model benefits, and captioning and retrieval performance.The study also compares the approach with state-of-the-art methods on both tasks.
- Visualization and Qualitative Analysis: Bidirectional LSTM gates progressively retain useful context and attenuate unimportant information, producing richer output representations over time.Input, forget, and output gates, together with cell states, are visualized across 11 time steps.
- Visualization and Qualitative Analysis: T-LSTM encodes past and future textual context, M-LSTM combines textual and CNN-encoded visual context, and Softmax predicts the next-word distribution.Higher-layer multimodal representations can develop sparse, discriminative patterns by removing information that contributes less to visual-textual correlation.
- Visualization and Qualitative Analysis: Generated forward and backward captions can cover different semantics, describe static scenes versus possible actions, and produce novel sentences similar to ground-truth captions.A random sample found 88% of generated sentences were novel relative to the training set on the MSCOCO validation set.
- Results on Caption Generation: On caption generation, deeper variants generally improve over Bi-LSTM, while VggNet substantially improves BLEU scores; small Flickr8K can disadvantage deeper models.Stacked LSTM layers are generally superior to fully connected transition layers, although Bi-S-L-LSTM requires more training time.
- Results on Caption Generation: 19.4/49.6 on Flickr8K and 16.2/28.2 on Flickr30K exceed DeepVSV’s 16.7/31.8 and 15.3/24.7 for METEOR/CIDEr, while MSCOCO reaches 20.8/66.6 versus 19.5/66.0.These comparisons are reported without object detection or a more powerful vision model.
- Results on Image-Sentence Retrieval: Retrieval performance exceeds compared methods on most metrics, but stronger vision models improve results and deeper models can overfit relatively small datasets.Data augmentation alleviates overfitting to some degree, while the authors anticipate further gains from more training examples.
- Discussion: Deep models have only slightly higher testing time while yielding significant improvements, and Bi-F-LSTM offers a balance between performance and efficiency.Testing costs include caption generation and image-to-sentence retrieval for ten Flickr8K validation images, averaged over five runs.
5. CONCLUSIONS
The paper concludes that bidirectional LSTM models generate image descriptions using both history and future context, while deep variants learn high-level visual-language embeddings. Across generation and retrieval evaluations, the models achieve highly competitive or state-of-the-art results.
- Bidirectional LSTMs generate descriptive image sentences by incorporating both historical and future context.
- Deep bidirectional LSTM architectures embed images and sentences in a high-level semantic space for visual-language modeling.
- Visualizing internal states qualitatively reveals how the multimodal LSTM generates words at consecutive time steps.
- Evaluations on numerous datasets assess the proposed models' effectiveness, generality, and robustness.
- The models achieve highly competitive or state-of-the-art results on both caption generation and image-sentence retrieval tasks.