Source-linked AI summary
From Captions to Visual Concepts and Back
Hao Fang, Saurabh Gupta, Forrest Iandola, Rupesh Srivastava, Li Deng, Piotr Dollár, Jianfeng Gao, Xiaodong He, Margaret Mitchell, John C. Platt, C. Lawrence Zitnick, Geoffrey Zweig
TL;DR
The paper addresses automatic generation of novel image descriptions from image-caption data without relying on separately labeled visual detectors. It learns caption-word detectors, conditions a maximum-entropy language model on those detections, and re-ranks candidates with multimodal similarity; on COCO, it reaches 29.1% BLEU-4 and matches or exceeds human quality 34% of the time.
Problem
The paper seeks to generate novel image captions while learning visual concepts and language statistics directly from image-caption pairs rather than separate hand-labeled detector data.
Method
The system uses Multiple Instance Learning for caption-word detectors, a maximum-entropy language model conditioned on detected words, and a deep multimodal similarity model for re-ranking.
Results
29.1% BLEU-4 versus 21.7% for humans on unseen COCO test data, while human judges rated system captions equal to or better than human captions 34% of the time.
Takeaways & Limitations
Learning visual detectors and language models directly from captions, together with global multimodal re-ranking, produced state-of-the-art COCO captioning results.
Takeaways & Limitations
The method cannot use standard supervised detector training for concepts whose image bounding boxes are not easily defined, such as open or beautiful.
Abstract
from arXiv · showhide
This paper presents a novel approach for automatically generating image descriptions: visual detectors, language models, and multimodal similarity models learnt directly from a dataset of image captions. We use multiple instance learning to train visual detectors for words that commonly occur in captions, including many different parts of speech such as nouns, verbs, and adjectives. The word detector outputs serve as conditional inputs to a maximum-entropy language model. The language model learns from a set of over 400,000 image descriptions to capture the statistics of word usage. We capture global semantics by re-ranking caption candidates using sentence-level features and a deep multimodal similarity model. Our system is state-of-the-art on the official Microsoft COCO benchmark, producing a BLEU-4 score of 29.1%. When human judges compare the system captions to ones written by other people on our held-out test set, the system captions have equal or better quality 34% of the time.
1. Introduction
The paper generates novel image captions by learning visual word detectors and language models directly from image-caption pairs, then re-ranking candidates with global multimodal similarity. Evaluated on Microsoft COCO, the system outperforms human captions on BLEU-4 and matches or exceeds human quality in a substantial fraction of judgments.
- Approach: The system detects caption words from image regions, including nouns, verbs, and adjectives, using weakly supervised learning directly from captions.This avoids reliance on bounding-box annotations and supports concepts that lack well-defined boxes.
- Approach: A maximum-entropy language model converts visual word detections into likely sentences using word-usage statistics learned from image descriptions.The model also captures commonsense knowledge that helps disambiguate noisy visual detections.
- Approach: A deep multimodal similarity model re-ranks high-likelihood caption candidates by measuring global compatibility between image and text.Its score is combined with other sentence features using learned linear weights.
- Evaluation: 29.1% BLEU-4 versus 21.7% for humans on unseen Microsoft COCO test data.The evaluation also used METEOR, perplexity, and human better/worse/equal judgments.
- Evaluation: 34% of the time, human judges rated system captions equal to or better than human-written captions on the held-out test set.The system also showed marked improvements over previous work on the PASCAL sentence dataset.
2. Related Work
Prior captioning systems either retrieve existing human-written descriptions or generate new ones using predefined visual and linguistic structures. This paper instead learns caption-relevant word detectors directly from captions and searches for likely sentences conditioned on those detections.
- Existing approaches: Retrieval methods produce well-formed human captions but may fail to describe novel combinations of objects or scenes.This limitation motivates generative captioning approaches.
- Existing approaches: Generative systems typically analyze images with object detectors before producing captions under syntactic or semantic constraints.The paper compares against prior work in this line, including Midge.
- Existing approaches: Neural and statistical alternatives condition language generation on image features, often using recurrent, log-bilinear, phrase-based, or attention models.These approaches differ in how visual information guides language generation.
- This paper: Unlike these approaches, the paper detects words from image regions with CNNs and Multiple Instance Learning while training directly from captions.This reduces assumptions about sentence structure.
- This paper: The generation task is formulated as searching for the most likely sentence rather than only conditioning a language model on image features.The search is conditioned on detected words.
3. Word Detection
The word-detection stage learns independent detectors for a caption vocabulary using weakly supervised Multiple Instance Learning over image regions. Detector outputs are calibrated and used to identify words and their spatial responses without bounding-box annotations.
- Vocabulary: The vocabulary contains the 1,000 most common caption words, covering over 92% of word occurrences in the training data.Words may belong to any part of speech, including nouns, verbs, and adjectives.
- Training Word Detectors: Multiple Instance Learning treats each image as a bag of regions, labeling bags positive when a caption contains the target word and negative otherwise.Training iteratively selects instances in positive bags and retrains the detector.
- Training Word Detectors: The noisy-OR MIL model combines region-level probabilities to estimate whether an image contains a target caption word.Each region probability is computed from CNN features using a logistic function.
- Localization: Fully convolutional CNNs produce coarse spatial response maps by scanning overlapping image regions for word-related visual patterns.The network uses a 12 × 12 response map at fc8 after up-sampling the image.
- Calibration: Detector outputs are calibrated using held-out precision and a global threshold, producing selected words with image-level probabilities and raw region scores.Detectors are trained independently, so calibration is needed before combining their outputs.
- Results: Without bounding-box annotations, the method still localizes objects reliably and associates image regions with more abstract concepts.Qualitative localization examples are shown in Figure 2.
4. Language Generation
The language-generation stage searches for likely sentences conditioned on visually detected words, using a maximum-entropy language model and beam search. Candidate captions are then assembled into an M-best list and re-ranked with sentence-level features.
- Statistical Model: The language model conditions each next-word probability on preceding words and detected words that remain unused in the sentence.This encourages detected words to be included while avoiding repetitions.
- Statistical Model: The baseline maximum-entropy model uses discrete features, with an added detector-score feature evaluating each word’s visual likelihood.Other features improved perplexity but not BLEU, METEOR, or human preference.
- Statistical Model: The model is trained by maximizing caption log-likelihood conditioned on the corresponding detected objects.Noise-contrastive estimation accelerates training by avoiding the exact denominator calculation.
- Generation Process: Left-to-right beam search retains the top k partial hypotheses while extending them with likely words, attributes, and valid continuations.Search stops when completed sentences are produced or the maximum sentence length is reached.
- Generation Process: Completed sentences covering at least a target number of image attributes form an M-best list, with the target reduced if too few candidates qualify.Candidates are initially sorted by log-likelihood.
5. Sentence Re-Ranking
The system re-ranks M-best caption candidates using sentence-level features, including a learned multimodal similarity score between images and text. The DMSM supplies this global similarity through jointly trained image and text networks mapped into a common semantic space.
- MERT re-ranks the M-best sentences using a linear combination of features computed over entire sentences, then selects the best sequence as the image caption.The model is trained on validation-set M-best lists using BLEU and applied to test-set lists.
- The DMSM maps images and text fragments into a common vector representation so their global similarity can guide sentence selection.This model replaces the query vector in DSSM with an image vector computed from a deep convolutional network.
- Image representations use a CNN-based model fine-tuned on COCO caption words, followed by additional fully connected layers with tanh nonlinearities.
- The text model represents captions using letter-trigram count vectors, which reduce input size and generalize to infrequent, unseen, and misspelled words.
- Relevance between an image and text is defined as cosine similarity between their learned representations, and this score is used by MERT to re-rank captions.
- Training computes posterior relevance using one matching caption and N randomly selected non-matching captions, with γ set to 10 and N set to 50.The model parameters are adjusted to minimize the negative log posterior probability of matching captions to images.
6. Experimental Results
Experiments evaluate word detection and caption generation on Microsoft COCO and PASCAL. Multiple-instance learning improves word prediction, while detector scores and multimodal re-ranking improve caption quality against baselines and human references.
- Word Detection: MIL NOR improves over chance and full-image classification for word detection across all parts of speech.The largest improvements occur for nouns and adjectives, which often correspond to concrete image subregions.
- Word Detection: Word detection is strongest for visually informative or object-associated words and weakest for less visually informative function words.Examples include polar at AP 94.6 and stuffed at AP 74.2, versus before at AP 1.0.
- Caption Generation: 25.69% vs. 19.32% BLEU: adding word-detector scores and DMSM re-ranking raises caption quality above human-generated captions on the held-out COCO test set.DMSM improvements with the VGG model are statistically significant for 4-gram overlap and per-image METEOR (p < .001).
- Caption Generation: 34% of the time, the final VGG+Score+DMSM+ft captions are judged the same or better than human-written descriptions.The comparison uses human studies on held-out COCO caption pairs.
- Caption Generation: 29.1% BLEU-4: on the official COCO test server, the system equals or surpasses human performance on 12 of 14 reported metrics.The system is reported as state-of-the-art on all 14 metrics among four publicly available competing results.
- Caption Generation: On the PASCAL sentence dataset, the approach improves over Midge from 2.0% to 17.6% BLEU and from 9.2% to 19.2% METEOR.The comparison uses 847 images tested by both systems.
7. Conclusion
The system generates novel captions by detecting words from image regions, using those detections to guide language generation, and re-ranking candidates with multimodal similarity. It achieves strong COCO benchmark performance and human-judged quality, although one comparison is excluded because a competing system produces multi-sentence captions.
- The system detects nouns, verbs, and adjectives from image regions, guides a language model with these words, and re-ranks candidate captions using global deep multimodal similarity.The pipeline combines visual word detection, language generation, and image-text similarity.
- State-of-the-art performance was reported on all 14 official COCO image-captioning metrics, matching or exceeding human performance on 12.
- 34% of human evaluations judged the generated captions equal to or better than human-written captions.
- Evaluation excludes Baby Talk because its long, multi-sentence captions make BLEU and METEOR comparisons difficult.