Source-linked AI summary
Learning Visual N-Grams from Web Data
Ang Li, Allan Jabri, Armand Joulin, Laurens van der Maaten
TL;DR
Real-world recognition needs broad visual vocabularies, but manually annotating images for every class does not scale. The paper trains image-conditioned visual n-gram convolutional models on web images and user comments, and demonstrates their use across phrase-based tasks and zero-shot transfer.
Problem
Manual annotation for thousands of images per class is infeasible when recognition systems must cover tens of thousands of visual concepts.
Method
The paper trains feed-forward convolutional visual n-gram models on web images and user comments using a differentiable smoothing loss inspired by language-model n-grams.
Results
The models demonstrate merits in image tagging, image retrieval, image captioning, and zero-shot transfer, with COCO captioning achieving a METEOR score of 17.2 versus 15.7 for nearest neighbors and 19.5 for a recurrent network.
Takeaways & Limitations
Visual n-gram models support recognition of a broad range of image-relevant phrases and transfer to new tasks through their close relation to standard image classification.
Takeaways & Limitations
The paper studies only one n-gram smoother and leaves comparisons with alternatives and broader open-world applications for future work.
Abstract
from arXiv · showhide
Real-world image recognition systems need to recognize tens of thousands of classes that constitute a plethora of visual concepts. The traditional approach of annotating thousands of images per class for training is infeasible in such a scenario, prompting the use of webly supervised data. This paper explores the training of image-recognition systems on large numbers of images and associated user comments. In particular, we develop visual n-gram models that can predict arbitrary phrases that are relevant to the content of an image. Our visual n-gram models are feed-forward convolutional networks trained using new loss functions that are inspired by n-gram models commonly used in language modeling. We demonstrate the merits of our models in phrase prediction, phrase-based image retrieval, relating images and captions, and zero-shot transfer.
1. Introduction
Traditional recognition benchmarks rely on manually annotated images for a small, fixed vocabulary, whereas real-world systems must handle many open-world concepts. This paper uses web images and user comments to train visual n-gram models that predict image-relevant phrases.
- 1. Introduction: Open-world recognition requires far more visual concepts than traditional benchmarks’ small, manually annotated class sets.Traditional datasets assume every image is annotated for each class and that many annotations define each class.
- 1. Introduction: Web photo collections provide abundant images with weak metadata such as tags, captions, comments, and geolocations, avoiding manual annotation at scale.The paper uses these signals as readily available information about image content.
- 1. Introduction: Visual n-gram models assign likelihoods to arbitrary image-relevant phrases using convolutional networks trained with a differentiable smoothing loss inspired by language-model n-grams.The loss emphasizes frequent n-grams and uses smaller sub-grams for infrequent ones.
- 1. Introduction: The paper contributes n-gram prediction, a differentiable smoothing layer, and an out-of-vocabulary solution, evaluating them in tagging, retrieval, captioning, and zero-shot transfer.These contributions target recognition beyond a fixed vocabulary of discrete classes.
2. Related Work
The paper connects visual n-grams to weakly supervised web learning, image-language modeling, and traditional language-model smoothing. It differs from related approaches by predicting phrases rather than only single words and by conditioning an n-gram model on images.
- Learning from weakly supervised web data: Prior work uses weakly labeled web images, but this paper avoids Google Images because retrieval may replicate an existing black-box vision system and introduce difficult-to-understand biases.The concern is specifically about biases introduced by the image-retrieval process.
- Learning from weakly supervised web data: Unlike prior visual models restricted to single words, this study predicts phrases and distinguishes expressions such as “city park” from “Park City”.The paper identifies phrase-level prediction as a key difference from [28].
- Relating image content and language: Unlike most image-captioning systems that use recurrent or maximum-entropy language models over convolutional features, this approach uses image-conditioned visual n-grams.The related work notes that recurrent networks are not considered because test-time inference is slow.
- Language models: The models adapt language-model smoothing techniques such as Jelinek-Mercer smoothing to parametric, image-conditioned visual recognition.Traditional n-gram language models and the paper’s models differ in whether they model language alone or condition predictions on images.
3. Learning Visual N-Gram Models
The paper trains visual n-gram models on images and associated comments, using convolutional networks and losses that predict both in-dictionary and out-of-dictionary phrases. Its method combines image-conditioned phrase distributions, smoothing, and scalable optimization over a large n-gram output space.
- The training data comprise 30 million English-comment examples selected from the 99.2-million-image YFCC100M dataset.Images are paired with multilingual user comments, filtered to English before training and testing.
- Loss functions: The smoothed n-gram loss assigns non-zero probabilities to out-of-dictionary phrases through Jelinek-Mercer probability-mass transfer.The smoothing constant recursively transfers mass from shorter n-grams to in-dictionary and out-of-dictionary n-grams, allowing learning from low-frequency phrases.
- Visual n-gram models assign likelihoods to phrases from image features and learn from image-comment pairs using convolutional networks.The predictive distribution is governed by image features and an n-gram embedding matrix.
- Loss functions: The naive loss sums logistic losses over in-dictionary n-grams appearing in each comment and ignores n-grams outside the dictionary.Its observational likelihood uses a softmax over in-dictionary n-grams based on image-feature and embedding inner products.
- Loss functions: Because the proposed smoothed loss is differentiable with respect to the embedding and network parameters, its gradient can be backpropagated through the convolutional network.The paper gives the loss gradient with respect to the image representation and the associated partial derivatives.
- Training: Training uses a 34-layer residual network and stochastic optimization over subsets of the 142,806 output columns to reduce the cost of forward-backward computation.The output-subset approximation includes the batch's positive n-grams and is described as closely related to the exact loss.
4. Experiments
Across four experiment sets, visual n-gram models are evaluated for phrase prediction, phrase-based retrieval, image-caption retrieval, and zero-shot transfer, with smoothing improving results over naive losses and baselines.
- 4. Experiments: Smoothing losses produce better visual n-gram models than naive n-gram losses, and additional test-time smoothing can further reduce perplexity.The smoothing loss directs learning toward the most relevant n-grams instead of spreading the signal equally across target n-grams.
- 4.1. Phrase-level image tagging: 42.0% of predicted phrases were judged relevant to image content by human annotators, supporting the phrase-prediction quality of the models.The evaluation used Mechanical Turk raters to assess whether the highest-scoring predicted n-gram matched the image content.
- 4.1. Phrase-level image tagging: Phrase-retrieval performance increases when longer n-grams are added, but improvements beyond trigrams are limited.Figure 2 compares models using dictionaries with maximum n-gram lengths from one to five.
- 4.3. Relating Images and Captions: The best model retrieved relevant captions for 53.1% of YFCC100M test images according to human raters.Caption retrieval was evaluated on 10,000 YFCC100M test images, alongside recall@k.
- 4.3. Relating Images and Captions: On COCO-5K and Flickr-30K, visual n-gram models perform roughly on par with state-of-the-art image-conditioned language models, while trailing dedicated retrieval models on Flickr-30K.The models match a state-of-the-art retrieval model on COCO-5K and use substantially larger vocabularies than the baselines.
5. Discussion and Future Work
Visual n-gram models offer an efficient alternative to recurrent vision-language systems and can support visual grounding and transfer, while remaining less suitable for caption generation. The paper also emphasizes learning from web data without manual annotation, while identifying benchmark bias and several directions for future work.
- Visual n-grams and recurrent models: Visual n-gram models are less suitable for caption generation but more efficient to evaluate at inference time than recurrent models.On COCO captioning, the model scores 17.2 METEOR versus 15.7 for nearest neighbors and 19.5 for a recurrent network.
- Visual n-grams and recurrent models: Class activation mapping enables visual grounding of n-grams, supporting transfer to new tasks through their connection to image classification.
- Learning from web data: Learning purely from web data avoids manual annotation but may not achieve state-of-the-art results on established benchmarks because those models do not exploit benchmark biases.
- Future work: Future work includes comparing additional n-gram smoothers, combining visual n-grams with zero-shot and few-shot learning, and applying them to broader vision-language tasks.
Supplementary Material for “Learning Visual N-Grams from Web Data”
The supplementary material lists the authors and their institutional affiliations.
- The paper lists Allan Jabri, Armand Joulin, and Laurens van der Maaten as authors from Facebook AI Research.
- The paper also lists the University of Maryland in College Park, Maryland.
1. Introduction
The supplementary material documents supporting information for the paper, including image licenses, COCO retrieval results, and additional phrase-prediction examples.
- The supplement provides license information for YFCC100M images used in the main paper.
- It reports quantitative image and caption retrieval results on the COCO-1K test set of 1,000 images.
- It presents additional qualitative results for phrase prediction.
2. License Information for YFCC100M Photos
The supplement reproduces YFCC100M photos appearing in the main paper together with authorship and license information.
- All YFCC100M photos appearing in the main paper are reproduced with relevant authorship and license information.
3. Relating Images and Captions: Additional Results
On COCO-1K, the visual n-gram model matches recurrent language models but trails retrieval-specific systems. End-to-end training also outperforms ImageNet-feature training on COCO-5K.
- The additional COCO-1K evaluation reports caption-retrieval and image-retrieval recall@k for baseline systems and visual n-gram variants.
- On COCO-1K, the visual n-gram model performs roughly on par with recurrent language models but worse than retrieval-specific models.
- Qualitative examples include high-scoring visual n-grams for test images, including phrases such as “Mardi Gras parade in progress” and “Formula 1 race for the”.
- End-to-end Jelinek-Mercer training outperforms ImageNet-feature training in both non-finetuning and finetuning modes on COCO-5K.
4. Phrase Prediction: Additional Results
Additional results examine retrieval performance, qualitative phrase predictions, and discriminative image regions. The reported qualitative examples include meaningful unigram and bigram predictions, while the retrieval tables organize recall@k across model variants and baselines.
- Figure 4 visualizes discriminative regions for five n-grams across three images using class activation mapping.
- Table 1 reports caption- and image-retrieval recall@k on COCO-1K for baseline systems and visual n-gram models with and without finetuning.
- Table 2 reports caption- and image-retrieval recall@k on COCO-5K for four visual n-gram variants with and without finetuning.
- Qualitative predictions include location, object, and scene phrases such as “Ferris wheel,” “Times Square,” “Golden Gate,” and “Horse drawn.”
- The qualitative results were omitted from the main paper because of space limitations.