Source-linked AI summary
Fast, Diverse and Accurate Image Captioning Guided By Part-of-Speech
Aditya Deshpande, Jyoti Aneja, Liwei Wang, Alexander Schwing, D. A. Forsyth
TL;DR
Image captioning must generate multiple accurate captions despite ambiguity, but beam search is slow and generic while alternative generative methods trade accuracy for diversity. The paper predicts quantized POS summaries from images and generates captions conditioned on them. It reports a trifecta of accurate, fast, and diverse captions relative to beam-search, VAE, and GAN-based methods.
Problem
Image captioning is ambiguous, and existing beam-search, VAE, and GAN approaches expose trade-offs among accuracy, speed, and diversity.
Method
The method predicts quantized POS tag sequences from an image and generates captions conditioned on each predicted summary.
Results
The POS-based approach achieves high accuracy, faster diverse-caption computation, and greater diversity than the compared beam-search, VAE, and GAN-based methods.
Takeaways & Limitations
POS tag sequences provide image summaries that steer caption generation toward diverse captions while retaining accuracy and computational efficiency.
Abstract
from arXiv · showhide
Image captioning is an ambiguous problem, with many suitable captions for an image. To address ambiguity, beam search is the de facto method for sampling multiple captions. However, beam search is computationally expensive and known to produce generic captions. To address this concern, some variational auto-encoder (VAE) and generative adversarial net (GAN) based methods have been proposed. Though diverse, GAN and VAE are less accurate. In this paper, we first predict a meaningful summary of the image, then generate the caption based on that summary. We use part-of-speech as summaries, since our summary should drive caption generation. We achieve the trifecta: (1) High accuracy for the diverse captions as evaluated by standard captioning metrics and user studies; (2) Faster computation of diverse captions compared to beam search and diverse beam search; and (3) High diversity as evaluated by counting novel sentences, distinct n-grams and mutual overlap (i.e., mBleu-4) scores.
1. Introduction
Image captioning must produce multiple captions that remain accurate despite ambiguity. The paper conditions caption generation on predicted POS summaries to obtain captions that are accurate, fast, and diverse.
- The method predicts different high-level POS summaries from an image and generates captions conditioned on each summary.
- Image captioning is ambiguous because many captions can describe the same image, creating a need for diverse captions that still describe the image properly.
- Beam search can generate accurate captions but requires wide beams for strong results, making inference slow and often producing generic captions.
- POS conditioning steers caption content, such as producing more or fewer adjectives, helping avoid minimal or generic captions.
- The paper reports that POS-based captioning is fast, diverse, and accurate compared with beam-search, VAE, and GAN-based approaches.
2. Related Work
Prior work improves captioning through architectures, training losses, object semantics, beam-search variants, VAEs, and GANs, but these approaches expose accuracy, speed, or diversity trade-offs. The paper positions POS tag sequences as global probes for sampling diverse captions.
- Earlier captioning systems commonly combine pretrained CNN image features with recurrent language models trained by maximum likelihood.
- Reinforcement-learning approaches directly optimize nondifferentiable metrics, whereas this paper uses maximum-likelihood training for fair comparisons.
- Object-based semantic conditioning focuses on nouns, while the POS approach coordinates semantic structure across the entire sentence.
- Beam search is accurate but slow and insufficiently diverse, while diverse beam search adds a Hamming-diversity penalty during local word-by-word expansion.
- GAN-based methods generally improve diversity but can lose accuracy, including reduced METEOR and SPICE scores relative to an LSTM baseline.
- The POS method reports higher best-1 and best-kth caption accuracy than AG-CVAE under the same ranking procedure and sampled-caption counts.
3. Background
The paper formulates diverse captioning as sampling and ranking multiple word sequences, then contrasts classical beam search with POS-guided inference. POS sampling expands captions using global tag-sequence probes rather than only local word probabilities.
- Problem Setup and Notation: Diverse captioning seeks k image-describing sequences, each composed of vocabulary words and associated with one of 12 universal POS tags.
- Training Data: Training data consists of image, caption, and POS-tag-sequence tuples, with POS labels obtained automatically because annotating roughly .5M MSCOCO captions is infeasible.
- Classical Image Captioning: Classical captioning factorizes pθ(y|I) into word-wise conditionals, so each word depends on the preceding words and image.
- Beam Search: Beam search maintains the best-k incomplete captions by likelihood and greedily expands them from the beginning to the end of the sentence.
- Inference Complexity: Choosing the maximum-probability word at each position yields O(k|Y|) computation, while classical expand-top-k operations require O(k^2 + |Y|k log k).
- POS Captioning: POS inference samples k quantized tag sequences, encodes each with an LSTM, and conditions temporal-convolution captioning on POS, image, object, and previous-word features.
4. Image Captioning with Part-of-Speech
The method conditions caption generation on quantized POS tag sequences, predicting image-specific summaries and then generating captions from each summary. It combines global sentence-structure control with efficient, diverse inference through narrow or greedy decoding.
- Image Captioning with Part-of-Speech: Quantized POS tag sequences condition the recurrent captioning model and provide global control over sentence structure.The full POS sequence is used in the conditional so it can control the entire sentence structure.
- Image Captioning with Part-of-Speech: At test time, the system samples k POS tag sequences and generates one caption conditioned on each sequence.The quantized sequences approximate possible image-specific POS sequences, enabling multiple captions for one image.
- Image Captioning with Part-of-Speech: POS-based inference has time complexity O(k|Y|) because it greedily selects the maximum-probability word at each position.This uses one max-operation per word position rather than breadth-first beam expansion.
- Image Captioning with Part-of-Speech: POS-based sampling is faster than wide-beam search and diverse beam search because its effective beam size is 1.Diverse beam search instead performs k sequential expand-top-k operations with an augmented diversity function.
- Image to Part-of-Speech Classification: The method quantizes more than 210K training POS sequences into 1024 cluster medoids using Hamming-distance k-medoids.More than 75% of clusters have average Hamming distance below 3, and each sequence is represented by its nearest quantized exemplar.
- Separate vs. Joint Training: POS+Joint jointly trains the POS classifier and captioning network, sampling 50 POS sequences per iteration and selecting the best-aligned one.The paper reports that this joint-training variant produces more accurate captions.
5. Results
Across MS COCO evaluations, POS-based captioning combines accuracy comparable to beam-search methods with faster sampling and stronger diversity than competing approaches. It also outperforms AG-CVAE and GAN-based baselines on reported accuracy measures and receives better user-study judgments.
- Best-1 Accuracy: POS and POS+Joint achieve captioning accuracy comparable to Beam Search and Div-BS while sampling more efficiently.Consensus re-ranking is used to select the best-1 caption, with shared image features and re-ranking parameters for fair comparison.
- Diversity: POS produces more novel sentences than competing methods and has the lowest mBleu-4, indicating higher diversity among generated captions.The diversity evaluation also reports 19.26 unique sentences out of 20 and 91.55 out of 100 for POS, while joint training reduces uniqueness because noisy POS sequences can yield repeated captions.
- Best-1 Accuracy: POS and POS+Joint outperform AG-CVAE on all reported consensus-re-ranking metrics, with POS+Joint especially improving over POS for 100 samples.The reported gains include CIDEr, Bleu-1, Bleu-2, METEOR, and SPICE, while other scores are comparable to the third decimal.
- Best-1 Accuracy: POS+Joint obtains higher METEOR and SPICE scores than the reported GAN method on the Karpathy et al. MS COCO split.On this split, POS+Joint also obtains a better SPICE score than beam search; the authors note accuracy may improve with ResNet-152 features.
- Human perception: In a user study with 123 participants, POS captions were preferred over Beam Search and AG-CVAE captions.Users selected the caption that best described the image without knowing which method generated it.
- Best-kth Accuracy: Across best-k evaluations, POS methods maintain accuracy comparable to beam search while AG-CVAE accuracy drops sharply on CIDEr and SPICE.The comparison covers best-kth scores for k = 1 to 10, where the kth caption is evaluated rather than only the top-ranked caption.
- Speed: POS-based methods are 5× faster than beam-search methods for sampling multiple captions.Their time complexity matches VAE or GAN sampling when the maximum-probability word is selected at each position.
6. Conclusion
The part-of-speech-conditioned approach achieves higher accuracy than GAN- and VAE-based methods, greater computational efficiency than classical beam search, and stronger performance on diversity metrics.
- POS-conditioned captioning obtains higher accuracy than GAN- and VAE-based methods.
- The approach is computationally more efficient than classical beam search.
- POS conditioning performs better than other methods on different diversity metrics.