Source-linked AI summary
Deep Reinforcement Learning-based Image Captioning with Embedding Reward
Zhou Ren, Xiaoyu Wang, Ning Zhang, Xutao Lv, Li-Jia Li
TL;DR
Image captioning must generate natural-language descriptions despite complex visual content and diverse valid descriptions. The paper formulates captioning as decision-making with policy and value networks, trained using actor-critic reinforcement learning and visual-semantic embedding rewards. On MS COCO, the framework achieves state-of-the-art performance across standard metrics, while training faces a large action space and failures on small visual details remain possible.
Problem
Image captioning requires understanding complex image content and expressing it through diverse natural-language descriptions, while prevailing methods use sequential recurrent prediction.
Method
The framework uses a policy network for local next-word guidance and a value network for global lookahead guidance, trained with actor-critic reinforcement learning and visual-semantic embedding rewards.
Results
The method achieves state-of-the-art performance on the MS COCO benchmark and outperforms existing approaches across BLEU, Meteor, Rouge, and CIDEr.
Takeaways & Limitations
Combining local policy confidence with global value evaluation enables caption generation to consider words that policy-only selection might assign low probability.
Takeaways & Limitations
Training is difficult because image captioning has an action space of approximately 10^3, and failures can occur when important visual content occupies small image regions.
Abstract
from arXiv · showhide
Image captioning is a challenging problem owing to the complexity in understanding the image content and diverse ways of describing it in natural language. Recent advances in deep neural networks have substantially improved the performance of this task. Most state-of-the-art approaches follow an encoder-decoder framework, which generates captions using a sequential recurrent prediction model. However, in this paper, we introduce a novel decision-making framework for image captioning. We utilize a "policy network" and a "value network" to collaboratively generate captions. The policy network serves as a local guidance by providing the confidence of predicting the next word according to the current state. Additionally, the value network serves as a global and lookahead guidance by evaluating all possible extensions of the current state. In essence, it adjusts the goal of predicting the correct words towards the goal of generating captions similar to the ground truth captions. We train both networks using an actor-critic reinforcement learning model, with a novel reward defined by visual-semantic embedding. Extensive experiments and analyses on the Microsoft COCO dataset show that the proposed framework outperforms state-of-the-art approaches across different evaluation metrics.
1. Introduction
Image captioning commonly uses sequential encoder-decoder prediction, but this paper proposes a decision-making framework that combines local word confidence with global lookahead evaluation. Actor-critic reinforcement learning with visual-semantic embedding reward trains the framework, which outperforms state-of-the-art methods across multiple metrics on MS COCO.
- Image captioning automatically describes image content in natural language, requiring machines to understand visual information and express it linguistically.
- Most state-of-the-art approaches encode images with convolutional networks and decode captions with recurrent networks by maximizing next-word probability.
- The proposed framework jointly uses a policy network for local next-word confidence and a value network for global lookahead evaluation.
- The value network shifts optimization from selecting individually correct words toward generating captions similar to ground-truth descriptions.
- The method outperforms state-of-the-art approaches consistently across BLEU, Meteor, Rouge, and CIDEr on Microsoft COCO.
- Actor-critic reinforcement learning driven by visual-semantic embedding trains the policy and value networks, with embedding supervision generalizing across evaluation metrics.
2. Related Work
Prior image-captioning research largely follows encoder-decoder models and locally guided greedy or beam decoding. This paper frames caption generation as decision-making and adds global lookahead guidance to reduce errors from locally preferred words.
- Earlier image-captioning methods generated descriptive words through object recognition and attribute prediction before combining them with language models.
- Recent encoder-decoder approaches encode images into feature vectors with CNNs and feed them into RNNs to generate captions.
- Most state-of-the-art decoders use greedy or beam search, sequentially selecting words according to local confidence.
- Local-confidence decoding can miss useful early words that would lead to better captions, motivating complementary global lookahead guidance.
- Decision-making research commonly models agents that execute action sequences to optimize predefined goals, including gaming, control, navigation, and path planning.
- Decision-making had not been applied to image captioning, while metric-driven REINFORCE training was difficult to generalize across evaluation metrics.
3. Deep Reinforcement Learning-based Image Captioning
The paper formulates image captioning as a decision-making problem in which policy and value networks jointly select words, using embedding-based rewards and actor-critic reinforcement learning. Lookahead inference combines local policy confidence with global value estimates to improve caption generation.
- 3.1. Formulation: Image captioning is formulated as a decision-making process where an agent generates a sentence by selecting words from a dictionary given an image.The state contains the image and words generated so far, while the policy and value networks constitute the agent.
- 3.1. Policy and value networks: The policy network provides local next-word probabilities, while the value network evaluates the expected reward of the current partially generated caption.The value network uses CNN, RNN, and MLP components to regress a scalar reward from the image and partial sentence.
- 3.2. Reward: Visual-semantic embedding similarity is used as the reinforcement-learning reward for generated sentences.The embedding model maps image and sentence representations into a shared semantic space, and the reward measures their similarity.
- 3.3. Training: The policy network is pretrained with cross-entropy loss and the value network with mean squared loss before joint deep reinforcement learning.The value-network regression target is the final reward of the generated sentence, with one randomly sampled state used per captioning process.
- 3.3. Training: The value network acts as a moving baseline, making the policy-gradient estimate lower variance while scaling updates by an advantage estimate.This training arrangement is described as an actor-critic architecture, with the policy as actor and value network as critic.
- 3.4. Lookahead inference: Lookahead inference scores candidate word extensions using both policy confidence and value-network evaluation, rather than relying only on local word probabilities.The method considers all single-word extensions and combines the two signals with hyperparameter λ, while curriculum learning addresses the large action space during training.
4. Experiments
Experiments on MS COCO evaluate the framework against existing methods, component variants, qualitative examples, value-network designs, and parameter sensitivity. The method achieves strong multi-metric performance, benefits from embedding-driven decision-making and lookahead guidance, and is comparatively robust to beam size.
- Comparison with state-of-the-art methods: The method achieves state-of-the-art performance on MS COCO in most evaluation metrics, using beam size 10 for comparison.Table 1 compares the method with existing approaches; competing results may use external training data.
- Component analysis: Embedding alone provides only a slight improvement over supervised learning, whereas the full embedding-driven decision-making framework performs much better.The ablation compares SL, SL-Embed, and Full-model variants.
- Component analysis: Lookahead inference improves the supervised baseline, and reinforcement learning further improves the model beyond supervised learning with a pretrained raw value network.These comparisons isolate the contributions of the value network, lookahead inference, and actor-critic training.
- Qualitative analysis: Qualitative results show better recognition of key objects and fewer incorrect-word errors than the supervised baseline, while two failure cases remain.The reported failures involve important visual contents occupying small image regions.
- Value-network architecture analysis: The value network is better trained independently with raw image and sentence inputs rather than directly reusing the policy hidden state.The architecture analysis compares variants that use policy representations with an independently trained value network.
- Parameter sensitivity analysis: Overall performance is best at λ = 0.4, declines when λ moves toward either 0 or 1, and varies less across beam sizes than the supervised baseline.λ combines policy and value guidance; the framework's local and global guidance is associated with greater beam-size robustness.
5. Conclusion
The paper presents a decision-making framework that combines policy and value networks with actor-critic reinforcement learning and visual-semantic embedding rewards for image captioning.
- The framework uses a policy network for local guidance and a value network for global, lookahead guidance during caption generation.The two networks jointly generate captions rather than relying solely on the previous encoder-decoder approach.
- Both networks are learned with an actor-critic reinforcement learning approach using visual-semantic embedding rewards.
- The proposed framework achieves state-of-the-art performance on a standard image-captioning benchmark.
- Future work includes improving network architectures and investigating reward designs based on other embedding measures.