Source-linked AI summary
More Grounded Image Captioning by Distilling Image-Text Matching Model
Yuanen Zhou, Meng Wang, Daqing Liu, Zhenzhen Hu, Hanwang Zhang
TL;DR
Image captioners can generate strong captions while grounding their visual attention poorly, and collecting region-word annotations for correction is expensive. The paper proposes POS-SCAN, distills it into captioners, and studies image-text matching scores as SCST rewards to improve grounding.
Problem
Existing image captioners have unsatisfactory attention grounding, while fine-grained region-word alignment annotations are expensive to collect.
Method
The paper enhances SCAN with Part-of-Speech information and distills the resulting POS-SCAN image-text matching model into captioners as an attention regularizer.
Results
POS-SCAN improves caption quality and attention accuracy without ground-truth attention supervision, while SCAN rewards improve grounding and retain standard metric scores.
Takeaways & Limitations
Image-text matching can provide weak grounding supervision and a practical captioning–grounding trade-off without extra inference overhead.
Abstract
from arXiv · showhide
Visual attention not only improves the performance of image captioners, but also serves as a visual interpretation to qualitatively measure the caption rationality and model transparency. Specifically, we expect that a captioner can fix its attentive gaze on the correct objects while generating the corresponding words. This ability is also known as grounded image captioning. However, the grounding accuracy of existing captioners is far from satisfactory. To improve the grounding accuracy while retaining the captioning quality, it is expensive to collect the word-region alignment as strong supervision. To this end, we propose a Part-of-Speech (POS) enhanced image-text matching model (SCAN \cite{lee2018stacked}): POS-SCAN, as the effective knowledge distillation for more grounded image captioning. The benefits are two-fold: 1) given a sentence and an image, POS-SCAN can ground the objects more accurately than SCAN; 2) POS-SCAN serves as a word-region alignment regularization for the captioner's visual attention module. By showing benchmark experimental results, we demonstrate that conventional image captioners equipped with POS-SCAN can significantly improve the grounding accuracy without strong supervision. Last but not the least, we explore the indispensable Self-Critical Sequence Training (SCST) \cite{Rennie_2017_CVPR} in the context of grounded image captioning and show that the image-text matching score can serve as a reward for more grounded captioning \footnote{https://github.com/YuanEZhou/Grounded-Image-Captioning}.
1. Introduction
Existing captioners can produce high-quality captions while grounding attention poorly, motivating weakly supervised alignment from image-text matching models. The paper proposes POS-SCAN to improve grounding without expensive region-word annotations and examines matching scores as SCST rewards.
- Existing captioners’ poor attention grounding can cause object hallucinations and gender discrimination, reducing caption rationality and explainability.
- The method distills an independently trained image-text matching model into a captioner as weak supervision for visual attention.Image-text matching is treated as a weaker but more objective grounding signal than caption-generation objectives.
- Original SCAN grounds less accurately than Up-Down, achieving 17.63% attention accuracy versus 19.83% for Up-Down.
- POS-SCAN keeps noun words when computing matching scores because non-visual words and visual relationship verbs can harm weakly supervised grounding.
- POS-SCAN improves caption quality and attention accuracy without region-word alignment annotations on Flickr30k Entities.It retains the image-text matching framework while using POS information to address grounding weaknesses.
- SCST with SCAN rewards more faithful captions while retaining standard metric scores, whereas POS-SCAN rewards improve grounding but significantly worsen standard metrics.
2. Related Work
Prior work improves captioning, image-text matching, visual grounding, and knowledge transfer through distinct approaches. This paper connects these areas by distilling an image-text matching teacher into an image-captioning student.
- Attention-based captioners operate over spatial grids, semantic metadata, or object regions, but quantitative studies find their attention correctness unsatisfactory.
- Grounded captioning methods often require expensive fine-grained region-word alignment annotations, motivating weaker supervision.
- Image-text matching methods include global and local alignment; this paper uses local alignment to connect image regions with sentence words.
- The paper frames knowledge distillation as transferring knowledge from an easier image-text matching teacher to an image-captioning student with a harder task.
3. Approach
The framework combines a neural caption generator with SCAN-based image-text matching, using POS-SCAN to guide visual attention and SCAN similarity to reward grounded captions. Training distills local region-word alignments into the captioner and applies self-critical reinforcement learning, while the matching model is removed during testing.
- 3. Approach: The framework combines a neural image caption generator with an image-text matching model, using the matcher as both an attention guider and a fine-grained rewarder.The two components remain fixed for matching-model pretraining and are integrated during caption-generator training.
- 3.1. Image-Text Matching Model: SCAN transforms image regions and sentence words into embeddings, computes local region-word similarities and attention-weighted representations, and summarizes them into a global image-text similarity score.SCAN is trained with triplet loss and hard negative mining.
- 3. Approach: At test time, the matching model is removed, leaving the captioning model to generate descriptive captions whose words and regions are better aligned.The training pipeline therefore adds no matching-model computation during inference.
- 3.1. Image-Text Matching Model: POS-SCAN restricts matching-score computation to noun words selected by a Part-of-Speech tagger, improving the matching model’s grounding for the downstream task.The formulation replaces the original similarity score with a POS-aware score using a noun indicator.
- 3.3. Learning to Generate More Grounded Captions: During cross-entropy training, the captioner’s attention weights β_t are regularized toward POS-SCAN attention weights α_t with KL divergence at visually groundable words, without alignment annotations.The alignment-supervised variant instead uses ground-truth region indicators, but the proposed setting distills the teacher attention.
- 3.3. Learning to Generate More Grounded Captions: During SCST fine-tuning, the reward combines CIDEr with image-text similarity, using a sampled caption’s reward relative to a greedily decoded baseline.The reward is defined as r(y1:n) = CIDEr(y1:n) + λ2S(I, y1:n).
4. Experiments
Experiments evaluate grounding and caption quality on Flickr30k Entities and MS-COCO using alignment metrics, ablations, benchmark comparisons, and qualitative examples. POS-SCAN supervision improves grounding without region-word annotations, while SCST effects depend on the reward.
- Quantitative Analysis: Figure 3 reports that appropriate region-word alignment supervision can improve both captioning metrics, including CIDEr and SPICE, and attention metrics, including F1all and F1loc.The figure studies the effect of λ1 on the Flickr30k Entities validation set.
- Quantitative Analysis: 28.58% attention accuracy for POS-SCAN exceeds 17.63% for SCAN and 19.83% for Up-Down on annotated object words.The evaluation uses ground-truth sentences on the Flickr30k Entities validation set.
- Quantitative Analysis: POS-SCAN supervision significantly improves attention evaluation without region-word annotations, whereas original SCAN supervision does not.This comparison is reported for the XE Pre-Train stage on Flickr30k Entities.
- Quantitative Analysis: SCST with CIDEr significantly improves caption quality but degrades grounding in most cases because CIDEr rewards n-gram consistency rather than visual semantic alignment.Adding SCAN as a reward improves SPICE and grounding in most cases, providing a trade-off against POS-SCAN reward.
- Quantitative Analysis: The final model outperforms state-of-the-art methods on both caption and attention evaluations without ground-truth attention supervision.Comparisons are reported on Flickr30k Entities, with additional results on MS-COCO.
- Qualitative Analysis: Qualitative examples show more grounded captions with POS-SCAN supervision, while failure cases include repetition, mis-recognition, and misassociation in complex contexts.Figure 4 visualizes attended regions and weight distributions; Figure 5 presents representative failures.
5. Conclusions
The paper shows that POS-SCAN can produce more grounded captions without grounding annotations, improving interpretability and transparency. Using SCAN as an SCST reward offers a practical balance between caption quality and grounding performance.
- POS-SCAN enables more grounded captions without grounding annotations by distilling an image-text matching model.
- This approach enhances the interpretability and transparency of existing image captioning models.
- Incorporating the SCAN image-text matching score as a reward provides a practical trade-off between caption quality and grounding performance.