Source-linked AI summary
Exploring Nearest Neighbor Approaches for Image Captioning
Jacob Devlin, Saurabh Gupta, Ross Girshick, Margaret Mitchell, C. Lawrence Zitnick
TL;DR
Image captioning research leaves open how important novel caption generation is when large datasets enable effective caption copying. The paper explores nearest-neighbor retrieval and consensus selection on MS COCO, finding strong automatic-metric performance but a significant human preference for novel captions. The authors therefore highlight a mismatch between automatic evaluation and human judgment.
Problem
The paper asks how important novel caption generation is for image captioning when large datasets may provide suitable captions to copy.
Method
The paper evaluates nearest-neighbor methods that retrieve similar training images, pool their captions, and select a consensus caption using lexical-similarity scores.
Results
Nearest-neighbor methods outperform many novel caption generation approaches on BLEU, METEOR, and CIDEr, while humans significantly prefer novel captions.
Takeaways & Limitations
Automatic metrics can favor borrowed captions even when human evaluations prefer captions generated anew.
Takeaways & Limitations
The study explores only simple nearest-neighbor baselines, while caption detail depends on the diversity of retrieved images and automatic metrics may not align with human judgment.
Abstract
from arXiv · showhide
We explore a variety of nearest neighbor baseline approaches for image captioning. These approaches find a set of nearest neighbor images in the training set from which a caption may be borrowed for the query image. We select a caption for the query image by finding the caption that best represents the "consensus" of the set of candidate captions gathered from the nearest neighbor images. When measured by automatic evaluation metrics on the MS COCO caption evaluation server, these approaches perform as well as many recent approaches that generate novel captions. However, human studies show that a method that generates novel captions is still preferred over the nearest neighbor approach.
1 INTRODUCTION
The paper examines nearest-neighbor image-captioning approaches that borrow captions from similar training images and select a consensus caption. These simple methods perform strongly on automatic metrics, but human studies still prefer novel caption generation.
- 1 INTRODUCTION: The paper investigates whether generating novel captions is important when evaluating automatic image captioning on MS COCO.The motivation is that large caption datasets may make copying an appropriate caption increasingly effective.
- 1 INTRODUCTION: The study provides a detailed exploration of nearest-neighbor approaches to understand captioning limitations and MS COCO dataset properties.The work expands prior nearest-neighbor captioning research as a baseline investigation.
- 1 INTRODUCTION: Nearest-neighbor captioning retrieves k similar images, pools their captions, and selects the candidate with the strongest consensus using BLEU or CIDEr.Similarity is computed with GIST, pretrained deep features, or features fine-tuned for caption generation.
- 1 INTRODUCTION: The simple nearest-neighbor approach outperforms many novel caption generation approaches on BLEU, METEOR, and CIDEr using the MS COCO testing dataset.Deep features, especially those fine-tuned for caption generation, are effective for finding images whose captions can be borrowed.
- 1 INTRODUCTION: Despite strong automatic-metric performance, a crowdsourced study finds that humans significantly prefer a system generating novel captions.The paper notes that further human studies are needed to compare nearest-neighbor and other generation-based approaches.
2 RELATED WORK
Early image-captioning work explored copying captions from visually similar images, while related research formalized image–caption ranking as an alternative evaluation and selection task.
- Caption copying: Early approaches generated captions by copying them from other images, often using nearest-neighbor retrieval.Methods used image and caption features capturing objects, actions, scenes, and other visual information.
- Caption copying: Nearest-neighbor methods varied in their visual representations, including object, stuff, people, scene, and GIST information.One approach used Kernel Canonical Correlation Analysis to map images and captions into a common space for nearest-caption retrieval.
- Caption ranking: Hodosh et al. popularized image–caption ranking, which ranks candidate captions by relevance to a given image.They argued that ranking correlated more closely with human judgment than novel-caption generation evaluated with automatic metrics.
- Caption ranking: Caption-ranking approaches could also rank training captions to select a caption for a query image.This connects ranking methods to caption-selection systems based on retrieved training examples.
3 APPROACH
The approach retrieves visually similar training images, pools their captions, and selects a consensus caption using lexical agreement. It examines feature spaces and robust consensus selection while highlighting the trade-off between caption specificity and neighbor diversity.
- 3 APPROACH: The method retrieves k nearest training images using GIST, pretrained fc7, or caption-task-fine-tuned fc7-fine features.The value of k is chosen separately for each feature set and typically ranges from 50–200.
- 3 APPROACH: Captions from the nearest images are pooled into a candidate set C, containing n = 5k captions for MS COCO.Each MS COCO image has five captions.
- 3 APPROACH: The consensus caption is the candidate with the highest average lexical similarity to the other captions, using BLEU or CIDEr-D.CIDEr emphasizes rarer, more descriptive 1-to-4-gram phrases through tf-idf weighting.
- 3 APPROACH: To reduce noise from outlier captions, the method can optimize agreement over a subset M of C rather than the full candidate set.This subset-based selection is interpreted as finding the centroid of a large caption cluster.
- 3 APPROACH: A caption selected from many visually similar images is intended to describe the query image adequately, but neighbor diversity favors generic captions while similarity permits specificity.Caption detail reflects a risk–reward trade-off: specific descriptions are more rewarding when correct but riskier when incorrect.
- 3 APPROACH: Deep feature spaces produce better nearest-neighbor matches than GIST, whose low-level representation does not capture high-level scene semantics.The cited comparison reports that fc7 and fc7-fine do significantly better than GIST.
4 RESULTS
The experiments examine how nearest-neighbor configuration, feature space, image similarity, and evaluation method affect captioning performance. Nearest-neighbor systems perform strongly on automatic metrics, but their advantage varies with image similarity and does not match human preferences.
- 4.1 The Number of Nearest Neighbors: For k < 20, BLEU scores are significantly worse, while scores for k > 60 are roughly similar.Randomly selecting a caption from one image yields BLEU 11.2.
- 4.1 The Number of Nearest Neighbors: High BLEU scores occur for m values between 50 and 200, whereas using all n captions performs worse.The result supports removing outlier captions when computing the consensus.
- 4.2 Different Feature Spaces: CIDEr-based consensus selection is slightly better than BLEU-based selection on CIDEr and METEOR, while BLEU optimization performs better on BLEU.The comparison is reported for the fc7-fine approach.
- 4.2 Different Feature Spaces: GIST performs poorly, while deeply learned fc7 and fc7-fine features perform significantly better; fc7-fine (BLEU) is comparable to ME+DMSM on BLEU.GIST does not capture the high-level scene semantics used by the task.
- 4.3 Generalization to Novel Images: Images most visually similar to training data achieve the highest BLEU scores across approaches; nearest-neighbor methods do better on highly similar images but worse on highly dissimilar ones than generation-based methods.The bins are formed by mean fc7-fine distance to the 50 nearest training images.
- 4.4 MS COCO Caption Test: Despite strong automatic BLEU and CIDEr performance, human evaluation significantly favors the generation-based approach over nearest-neighbor systems.The paper uses repeated human judgments comparing produced captions with human-written captions.
5 DISCUSSION
The discussion highlights weaknesses in current image-captioning evaluation and identifies directions for testing generalization, refining human studies, and extending nearest-neighbor methods.
- Nearest-neighbor success exposes the need for testing datasets that measure generalization beyond training-set images.Suggested designs include evaluating unusual or diverse images, or collecting test queries from a different distribution than training images.
- CIDEr-selected consensus captions are more detailed than BLEU-selected captions in the illustrated examples.
- Automatic metrics may not properly benchmark progress toward humanlike captions because human captions can score lower than automatically generated captions.The authors call for metrics more highly correlated with human judgment.
- Overall human preference comparisons miss differences in fluency, content capture, and genericity between nearest-neighbor and novel-generation systems.Nearest-neighbor captions are described as fluent but potentially generic, while generated captions may capture content yet sometimes lack fluency.
- The study explores only simple nearest-neighbor baselines, leaving more sophisticated caption-ranking and hybrid retrieval-generation approaches for future work.A proposed hybrid would use nearest neighbors for queries with many similar training images and generation for others.