Source-linked AI summary
Stack-Captioning: Coarse-to-Fine Learning for Image Captioning
Jiuxiang Gu, Jianfei Cai, Gang Wang, Tsuhan Chen
TL;DR
One-stage captioning models struggle with rich fine-grained descriptions, while multi-stage models risk vanishing gradients. The paper introduces stacked coarse-to-fine decoders with intermediate supervision and normalized reinforcement-learning rewards, achieving comparable performance with an ensemble state-of-the-art approach on the online MSCOCO test server.
Problem
One-stage image-captioning models struggle to generate rich fine-grained descriptions, while multi-stage models face vanishing-gradient difficulties during training.
Method
The model stacks coarse and fine attention-based sentence decoders, trains them with intermediate supervision, and normalizes reinforcement-learning rewards using test-time inference and preceding-stage outputs.
Results
Comparable performance with the state-of-the-art ensemble approach was achieved on the online MSCOCO test server.
Takeaways & Limitations
Coarse-to-fine caption generation with stacked visual attention and multiple LSTM networks was beneficial for image captioning.
Abstract
from arXiv · showhide
The existing image captioning approaches typically train a one-stage sentence decoder, which is difficult to generate rich fine-grained descriptions. On the other hand, multi-stage image caption model is hard to train due to the vanishing gradient problem. In this paper, we propose a coarse-to-fine multi-stage prediction framework for image captioning, composed of multiple decoders each of which operates on the output of the previous stage, producing increasingly refined image descriptions. Our proposed learning approach addresses the difficulty of vanishing gradients during training by providing a learning objective function that enforces intermediate supervisions. Particularly, we optimize our model with a reinforcement learning approach which utilizes the output of each intermediate decoder's test-time inference algorithm as well as the output of its preceding decoder to normalize the rewards, which simultaneously solves the well-known exposure bias problem and the loss-evaluation mismatch problem. We extensively evaluate the proposed approach on MSCOCO and show that our approach can achieve the state-of-the-art performance.
Introduction
Image captioning seeks richer, more human-like descriptions, but one-stage models face difficulties with fine-grained generation, vanishing gradients, exposure bias, and loss-evaluation mismatch. The paper proposes a coarse-to-fine multi-stage framework with intermediate supervision and normalized reinforcement-learning rewards.
- Introduction: Rich fine-grained descriptions require high-complexity models, where back-propagated gradients can diminish through many layers.
- Introduction: The framework combines one CNN image encoder with attention-based LSTM decoders that refine descriptions from coarse to fine.
- Introduction: Training on previous ground-truth words differs from test-time generation from model predictions, causing exposure bias and error accumulation.
- Introduction: Cross-entropy training at each time step mismatches test-time sentence-level evaluation metrics such as BLEU-n, CIDEr, and SPICE.
- Introduction: The proposed model stacks sentence decoders that repeatedly refine image descriptions, while intermediate supervisions address vanishing gradients during multi-stage training.
- Introduction: Reinforcement learning introduces normalized intermediate rewards to optimize the multi-stage model and address exposure bias and loss-evaluation mismatch.
Related Works
Earlier image-captioning work developed CNN–RNN encoding-decoding models, visual attention, discriminative visual information, and reinforcement-learning objectives. This paper extends the predominantly one-stage paradigm with stacked coarse-to-fine decoders and stage-wise sentence-level supervision.
- Image Captioning with Maximum Likelihood Estimation: Classical image captioning uses a CNN image encoder and an RNN sentence decoder to connect visual content with descriptions.
- Image Captioning with Maximum Likelihood Estimation: Visual attention incorporates spatial attention over convolutional image features, improving the integration of image information into caption generation.
- Image Captioning with Maximum Likelihood Estimation: Later methods encode more discriminative visual information through attributes, including weakly supervised attribute detection and sequential attention mechanisms.
- Image Captioning with Reinforcement Learning: Reinforcement-learning approaches address discrepancies between training and testing objectives by optimizing sentence-level rewards for image captioning.
- Coarse-to-Fine Image Captioning: Unlike predominantly one-stage prior work, the proposed model stacks intermediate sentence decoders and uses sentence-level metrics across coarse and fine stages.
- Coarse-to-Fine Image Captioning: The coarse decoder produces an initial caption and reduces computational burden for fine-scale decoders generating complex descriptions.
Methodology
The model uses a coarse decoder followed by attention-based fine decoders that iteratively refine captions and visual attention. Intermediate cross-entropy supervision and reinforcement learning with relative rewards address training depth and evaluation-mode mismatches.
- Image representation: The model encodes an image into fixed-size spatial CNN features, with each feature channel representing an image region.Spatial adaptive average pooling produces a k × k × dv representation.
- Coarse-to-fine decoder: The architecture begins with one coarse decoder and sequential fine decoders, with each stage refining the preceding stage's image description.The final decoder's prediction is used as the final image description.
- Coarse-to-fine decoder: The coarse decoder predicts from global image features, while fine decoders use image features and preceding-stage attention weights to focus on detailed regions.Stacked attention progressively filters irrelevant visual information and pinpoints regions relevant to each predicted word.
- Fine decoder and attention: Each fine decoder combines attended image features, the previous word embedding, and its previous hidden state within an LSTM-based attention model.The attention function supplies attended visual representations as additional inputs to the fine LSTM.
- Intermediate supervision: To mitigate vanishing gradients, the network adds a cross-entropy objective at every decoder stage and sums the stage losses for the full architecture.Each stage repeatedly predicts the target words using parameters up to that stage.
- Reinforcement learning: Reinforcement learning uses relative rewards against both each stage's greedy test-time output and its preceding decoder's sampled output.The first baseline favors samples exceeding greedy decoding, while the second favors samples outperforming the previous stage.
Experiments
Experiments on MSCOCO evaluate Stack-Cap against established captioning baselines using standard offline and online benchmarks. The model achieves the best reported performance across metrics on the Karpathy test split, while qualitative analyses show progressively refined captions and attention.
- Datasets and Setting: MSCOCO evaluation uses BLEU, METEOR, ROUGE, CIDEr, and SPICE on offline splits, plus online comparison on a 40,775-image test set.The offline validation and test splits each contain 5,000 images; each MSCOCO image has five reference captions.
- Ablation and Analysis: Intermediate supervision prevents the overfitting observed when additional LSTM layers are added directly, and stacked attention further improves descriptions.Stack-Cap and Stack-Cap* perform similarly, while Stack-Cap with stacked attention outperforms Stack-Cap* by progressively adjusting attention toward relevant visual clues.
- Quantitative Results: Stack-Cap achieves the best performance in all metrics on the MSCOCO Karpathy test split.Table 3 compares Stack-Cap (C2F) with existing methods, while the reported metrics include BLEU, METEOR, ROUGE, CIDEr, and SPICE.
- Online Evaluation: The single Stack-Cap model achieves comparable performance with state-of-the-art approaches on all reported metrics in the online MSCOCO evaluation.The comparison notes that SCST:Att2in results use an ensemble of four models, whereas Stack-Cap uses a single model.
- Qualitative Analysis: Progressive attention filtering helps later stages identify finer content, such as both a dog and an umbrella, while aligning attention with relevant image regions.Qualitative visualizations show stage-wise refinement and strong alignment between attention maps and human-intuitive visual regions, including jets and their smoke trails.
Conclusion
The paper presents a coarse-to-fine captioning model that combines stacked visual attention with multiple LSTM networks, achieving comparable performance to an ensemble state-of-the-art approach on MSCOCO.
- The model combines stacked visual attention with multiple LSTM networks to generate progressively refined image descriptions.
- The coarse-to-fine generation strategy is reported as beneficial for image captioning compared with conventional one-stage models.
- The model achieves comparable performance with the state-of-the-art approach using an ensemble on the online MSCOCO test server.
- Future work includes integrating extra attribute learning and incorporating beam search into training.