Source-linked AI summary
An Empirical Study of Language CNN for Image Captioning
Jiuxiang Gu, Gang Wang, Jianfei Cai, Tsuhan Chen
TL;DR
Image captioning requires models to express image content in natural language while capturing long-range word dependencies. The paper introduces a language CNN that processes previous words hierarchically and combines it with recurrent networks for next-word prediction. Experiments on MS COCO and Flickr30K show improvements over recurrent counterparts and competitiveness with state-of-the-art methods.
Problem
Recurrent image-captioning models have difficulty retaining long-term information and explicitly modeling the hierarchical structure of word sequences.
Method
The paper combines a temporal language CNN for hierarchical and long-range word representations with recurrent networks for dynamic sequence prediction.
Results
Experiments on MS COCO and Flickr30K show that language-CNN models outperform recurrent-network counterparts across reported metrics and achieve competitive state-of-the-art performance.
Takeaways & Limitations
Language CNNs provide a useful complement to recurrent networks for modeling hierarchical and long-range information in image-captioning sequences.
Takeaways & Limitations
The models can predict incorrect visual attributes, reducing precision-based scores; the authors suggest incorporating extra attribute learning.
Abstract
from arXiv · showhide
Language Models based on recurrent neural networks have dominated recent image caption generation tasks. In this paper, we introduce a Language CNN model which is suitable for statistical language modeling tasks and shows competitive performance in image captioning. In contrast to previous models which predict next word based on one previous word and hidden state, our language CNN is fed with all the previous words and can model the long-range dependencies of history words, which are critical for image captioning. The effectiveness of our approach is validated on two datasets MS COCO and Flickr30K. Our extensive experimental results show that our method outperforms the vanilla recurrent neural network based language models and is competitive with the state-of-the-art methods.
1. Introduction
Image captioning requires models to understand visual content, semantic relationships, and natural language generation. The paper addresses limits of recurrent models by combining language CNNs with recurrent networks to capture hierarchical and long-range word dependencies.
- Image captioning is challenging because machines must understand image content, express relationships naturally, and capture implicit semantic information.
- RNNs may retain only a few time steps because of vanishing gradients, while LSTMs use memory cells and gates to learn longer dependencies.
- LSTM histories can still be diluted over time, and vanilla recurrent models do not explicitly represent the bottom-up hierarchy of word sequences.
- The proposed language CNN applies temporal convolution to model hierarchical and long-range dependencies, then combines it with recurrent networks to retain dynamic temporal behavior.
- The combined language-CNN and RNN model achieves comparable performance with state-of-the-art approaches on Flickr30K and incorporates long-range sequence modeling for image captioning.
2. Related Works
Earlier image-captioning systems used retrieval or encoder-decoder recurrent models, while later work incorporated attention and visual attributes. The paper identifies persistent difficulties in recurrent modeling of long-term and hierarchical word dependencies.
- Retrieval-based captioning cannot generate proper captions for new combinations of objects, motivating neural encoder-decoder approaches.
- Attention-based methods model correlations between image regions and descriptions through hard or soft spatial attention and review mechanisms.
- Other systems improve captioning by injecting high-level semantic attributes or classifier outputs into encoder-decoder models.
- Although recurrent networks model word sequences effectively, their repeatedly updated hidden states make long-term memory difficult and do not efficiently capture hierarchical word structure.
3.1. Overall Framework
The framework encodes the image with CNNI and previous words with CNNL, fuses both representations, and uses a recurrent network to predict each next word. Shared weights allow the process to operate across time frames.
- CNNI extracts VGGNet image features, while CNNL converts the generated-word sequence into a bottom-up representation before fusion with the image features.
- The multimodal representation and recurrent state determine the probability distribution of the next dictionary word through a Softmax layer.
- The model combines image encoding, sentence modeling, multimodal fusion, and recurrent word prediction in one framework.
- The framework estimates the next word from all previous words and the image, with weights shared across time frames.
3.2. CNNL Layer
CNNL uses word embeddings and temporal convolutions to build hierarchical sentence representations without pooling. Its fixed-length input uses padding or truncation, and deeper layers compose information across word positions.
- CNNL addresses the difficulty of explicitly modeling hierarchical word representations that remains in recurrent and multilayer LSTM models.
- The model embeds one-hot words into K-dimensional vectors and concatenates them as the input sequence representation.
- Temporal convolution is applied through a fixed maximum input length, with the practical CNNL using LL = 16 words.
- The network pads shorter sequences and truncates longer histories to fit its fixed input length.
- CNNL omits pooling because max-pooling would discard temporal information needed to model word composition.
- Convolutional layers build a hierarchical representation of history words that retains information useful for next-word prediction.
3.3. Multimodal Fusion Layer
The multimodal fusion layer combines language-CNN word representations with image features in a shared feature space to produce multimodal activations.
- The layer takes bottom-up word representations from CNNL and image representations from CNNI as its two inputs.
- It linearly maps both inputs into the same multimodal feature space and combines them to obtain the activation vector m[t].The combination uses element-wise addition, followed by a scaled tanh activation.
3.4. Recurrent Networks
The model supplements the language CNN with recurrent networks because holistic convolutional features may miss temporal information, and it evaluates several recurrent transition designs.
- The language CNN is combined with recurrent networks to compensate for its potential loss of important temporal information.CNNL extracts holistic features from the whole word sequence, which can miss temporal behavior.
- The recurrent component uses the previous recurrent state, previous word embedding, and multimodal fusion output to compute the next recurrent state.
- The study combines CNNL with Simple RNN, LSTM, GRU, and Recurrent Highway Network architectures.
- Simple RNNs are difficult to use for long-term dependencies because vanishing gradients make long-term directions smaller than short-term ones.
- LSTM controls information flow with gates and a memory cell, while GRU uses a simpler structure without a separate memory cell.The text states that GRU exposes its hidden state without control and is computationally more efficient.
- RHN introduces highway connections with directly gated links between the previous state and current input to modulate information flow.RHN lacks a reset gate, while CNNL can provide relevant history information at each prediction step.
3.5. Training
Training minimizes the summed negative log-likelihood of ground-truth caption words conditioned on preceding words and the image.
- The training instance consists of ground-truth words S paired with the corresponding image I.
- The loss for a training pair is the sum of negative log-likelihoods for the words in its ground-truth sentence.The sequence length is N, and S[t] denotes a word in sentence S.
- The objective maximizes the probability of each ground-truth context word given the image and preceding words.The word probabilities are produced by the Softmax layer, with θ denoting the model parameters.
3.6. Implementation Details
Implementation uses VGGNet image features, bounded caption preprocessing, embedded words, and fixed optimization and decoding settings across the evaluated datasets.
- Image features are computed with 16-layer VGGNet and mapped from the final fully connected layer into an embedding space by linear transformation.
- Captions are lowercased, stripped of nonalphabetic characters, converted to <UNK> for words occurring fewer than five times, and truncated at 16 tokens.The maximum CNNL input length is also set to 16 words.
- Each word representation is mapped as x[t] = WeS[t], with special <START> and <END> tokens marking sentence boundaries.The network predicts words after seeing the image and preceding words.
- Image features and word embeddings use dimensionality 512, while Adam optimization uses learning rates 2e-4 for Flickr30K and 4e-4 for MS COCO.Dropout and early stopping are used to reduce overfitting.
- At test time, the previous generated output replaces the ground-truth input, and beam search uses a fixed size k=2.
4. Experiments
Experiments evaluate CNNL-based captioning models against recurrent baselines and state-of-the-art systems on MS COCO and Flickr30K. CNNL improves recurrent counterparts across metrics, benefits from broader context without pooling, and produces more descriptive captions, while visual-attribute errors remain a limitation.
- 4.3.1 Analysis of CNNL: CNNL+RNN outperforms comparable LSTM models despite similar parameter scale, whereas deeper LSTMs overfit, especially on the smaller Flickr30K dataset.CNNL without recurrent layers performs worse, indicating recurrence remains useful for short-term contextual information.
- 4.3.1 Analysis of CNNL: Larger CNNL context windows improve performance, while max-pooling reduces performance by losing local word-order information.The experiments compare different history lengths and pooling variants on MS COCO.
- 4.3.2 Results Using CNNL on MS COCO: CNNL-based models outperform recurrent-network counterparts across all reported MS COCO metrics.CNNL+RHN leads B@(3,4), METEOR, and SPICE; CNNL+LSTM leads CIDEr at 99.1; CNNL+GRU leads B@(1,2).
- 4.3.3 Results Using CNNL on Flickr30K: On Flickr30K, CNNL+RHN leads all BLEU metrics, while CNNL+RNN leads METEOR, CIDEr, and SPICE.CNNL-based recurrent models also outperform their non-CNNL counterparts and are reported as more robust with less training data.
- 4.3.4 Comparison with State-of-the-art Methods: Compared with most state-of-the-art systems, CNNL models perform better on MS COCO and CNNL+RHN outperforms Attributes-CNN+RNN on Flickr30K.Attributes-CNN+RNN and ensemble-based Google NICv2 exceed the authors’ models on some MS COCO metrics; the authors use VGG-16 for comparison fairness.
- 4.4. Qualitative Results: Qualitatively, CNNL models generate more precise, descriptive, and humanlike captions by jointly using image representations and history words.Examples include improved descriptions of object relationships and visual content.
5. Conclusion
The paper presents an image captioning model that combines language CNNs with recurrent networks to capture hierarchical and temporal information in word sequences. Experiments on MS COCO and Flickr30K validate the proposal and show performance improvements over other image captioning methods.
- Combining language CNNs with recurrent networks lets the model explore both hierarchical and temporal information for image caption generation.