Source-linked AI summary
Training and Evaluating Multimodal Word Embeddings with Large-scale Web Annotated Images
Junhua Mao, Jiajing Xu, Yushi Jing, Alan Yuille
TL;DR
Word embeddings need large training corpora, while existing multimodal models and similarity benchmarks are constrained in scale and coverage. This paper introduces image-text and click-derived evaluation datasets and compares multimodal RNNs, reporting benefits from visual information and weight sharing in Model A.
Problem
Many embedding models use text alone, and existing word-similarity datasets have fewer than a thousand pairs and do not comprehensively evaluate training-set words.
Source: §1 · pp. 1–2Method
The paper builds a large Pinterest image-text dataset and a crowdsourced, click-derived evaluation set, then compares RNN models that combine text and images.
Source: §1 · pp. 1–2Results
Visual information significantly helps embeddings: Model A beats Word2Vec by 9.5% on Gold RP10K and 9.2% on Gold RP10M; sharing was similar for Models B and C.
Source: §5.3 · p. 8, §6 · p. 8Takeaways & Limitations
The triplet evaluation tests whether embeddings place a closer phrase nearer than a less-related phrase across varied semantic relationships.
Source: §3.2 · p. 4, §3.2.2 · p. 6The authors plan to separate similar from related phrase pairs and add pair scores to Gold RP10K, and to improve phrase representations.
Source: §6 · p. 8
Abstract
from arXiv · showhide
In this paper, we focus on training and evaluating effective word embeddings with both text and visual information. More specifically, we introduce a large-scale dataset with 300 million sentences describing over 40 million images crawled and downloaded from publicly available Pins (i.e. an image with sentence descriptions uploaded by users) on Pinterest. This dataset is more than 200 times larger than MS COCO, the standard large-scale image dataset with sentence descriptions. In addition, we construct an evaluation dataset to directly assess the effectiveness of word embeddings in terms of finding semantically similar or related words and phrases. The word/phrase pairs in this evaluation dataset are collected from the click data with millions of users in an image search system, thus contain rich semantic relationships. Based on these datasets, we propose and compare several Recurrent Neural Networks (RNNs) based multimodal (text and image) models. Experiments show that our model benefits from incorporating the visual information into the word embeddings, and a weight sharing strategy is crucial for learning such multimodal embeddings. The project page is: http://www.stat.ucla.edu/~junhua.mao/multimodal_embedding.html
1 Introduction
The paper addresses limited visual supervision in word embeddings and evaluation sets too small to comprehensively test learned representations. It introduces large-scale image-text and word/phrase evaluation datasets, then compares multimodal RNN models.
1 Introduction
- Existing multimodal models often use small datasets or visual constraints on selected concepts, while standard similarity benchmarks have fewer than a thousand pairs and miss many training-set words.The paper also notes that neural language-model training loss does not always correlate with embedding effectiveness on similarity or relatedness tasks.Source: §1 · pp. 1–2
- 40 million images and 300 million sentences make Pinterest40M comparable in scale to pure-text training corpora, versus 0.2 million images and 1 million sentences in MS COCO.The dataset is described as more than 200 times larger than the current MS COCO release.Source: §1 · p. 1
- Pinterest search clicks supplied millions of candidate word/phrase pairs, which crowdsourced annotations filtered into a 10,674-entry gold-standard evaluation set.The set targets evaluation of word and phrase similarity or relatedness.Source: §1 · p. 2
- The authors compare multimodal RNNs, with their best-performing model using weight sharing to softly constrain visual features against all related words in sentences.The paper presents this as a way to fuse visual information into learned embeddings.Source: §1 · p. 2
2 Related Work
Related work covers small image-description datasets, existing word-similarity tests, and approaches to multimodal embeddings. The paper positions Pinterest40M and its weight-sharing model against limitations in dataset scale and visual supervision.
Image-Sentence Description Datasets
- Image-description datasets supported vision-language research, but MS COCO had 1 million sentences versus Pinterest40M’s 300 million, and the existing descriptions used relatively simple language.Pinterest40M lists 335,323 unique words appearing at least 50 times, compared with around 10,000 words appearing at least three times in MS COCO.Source: §2 · p. 2
- YFCC100M supplies complementary Flickr metadata, but its users’ comments differ from Pinterest descriptions, including greater discussion of photography techniques.The passage attributes the difference to the platforms’ different functionality and user groups.Source: §2 · p. 2
Word Similarity-Relatedness Evaluation
- Standard similarity benchmarks contain only hundreds of pairs, while analogy questions do not directly measure similarity or relatedness and cannot cover millions of dictionary words.Pinterest search clicks provide another source of word/phrase pairs and are suited to multimodal embedding evaluation.Source: §2 · pp. 2–3
RNN for Language and Vision
- The paper’s language-and-vision models draw on CNN-RNN image-captioning work and use gated recurrent units, a variation of simple RNNs.The cited captioning models are described as a special case of sequence-to-sequence learning.Source: §2 · p. 3
Multimodal Word Embedding Models
- Prior multimodal methods either fuse separately extracted features or jointly associate visuals with selected concepts; the paper’s best model softly constrains visual features against all sentence words.The stated contrast is tied to the lack of large-scale multimodal datasets in prior work.Source: §2 · p. 3
3 Datasets
The paper introduces Pinterest40M for multimodal training and click-derived phrase triplets for evaluating word and phrase embeddings. It describes how the training corpus is assembled and how raw triplets are created and cleaned into a gold-standard dataset.
3.1 Training Dataset
- The Pinterest training corpus is assembled from public images and their user descriptions, with short and duplicate sentences removed.Each image has an average of 12 sentences before filtering.Source: §3.1 · pp. 3–4
- Pinterest40M contains 40 million images, 300 million sentences, and around 3 billion words, with richer user descriptions and a 335,323-word vocabulary at a minimum frequency of 50.The paper describes it as much larger than previous image-description datasets and reports no earlier multimodal RNN training dataset at this scale.Source: §3.1 · p. 4
3.2 Evaluation Datasets
- The evaluation uses labeled phrase triplets, where phrase A is semantically closer to phrase B than to phrase C, and marks a test triplet positive when d(A, B) < d(A, C).The dataset contains 9.8 million triplets generated from Pinterest image-search clicks and a cleaned gold standard of 10 thousand triplets.Source: §3.2 · p. 4
3.2.1 The Raw Evaluation Dataset from User Clickthrough Data
- For the raw evaluation set, click frequencies rank annotations as positive phrases for a query, after which phrases sharing query words or word roots are removed.Negative phrases are sampled from a pool of 1 billion phrases and must not overlap either phrase in the original pair.Source: §3.2.1 · pp. 4–5
3.2.2 The Cleaned-up Gold Standard Dataset
- Crowdsourcing addresses noisy click-derived triplets: annotators compare each positive and negative phrase with the base phrase, producing a gold standard of 10,674 triplets.A triplet is accepted only when more than half of its annotators agree with the original labels.Source: §3.2.2 · pp. 5–6
4 The Multimodal Word Embedding Models
The models combine CNN image features with an RNN language model, but differ in how they fuse visual information. Model A initializes the GRU from the image and uses transposed weight sharing; Models B and C add direct visual-supervision losses.
4 The Multimodal Word Embedding Models
- All models use a CNN to extract image representations and an RNN to model sentences; the RNN takes words as input and predicts the next word.The visual features are mapped to the GRU-state space in Models A and B, or the word-embedding space in Model C.Source: §4 · p. 6
- The GRU uses a 512-dimensional state, while training uses sampled SoftMax with 1,024 negative words for each step.Words occurring more than 50 times are included in a vocabulary of 335,323 words.Source: §4 · p. 6
- Model A initializes the GRU state from the visual representation and uses transposed weight sharing between the word-embedding and SoftMax matrices.The shared weights encourage visual information decoded by the output layer to enter the word embeddings.Source: §4 · pp. 6–7
- Models B and C add visual-supervision losses on the final GRU state and word embeddings, respectively, alongside the sampled SoftMax loss.These added losses are balanced against the language-model loss by a weight hyperparameter.Source: §4 · p. 7
5 Experiments
Experiments compare multimodal RNN models and text baselines on RP10M and Gold RP10K. Model A performs best among the proposed models, with reported gains over Word2Vec and the variant without weight sharing.
5.1 Training Details; 5.2 Evaluation Details
- Evaluation reports average precision over phrase triplets; training used mini-batches of 256 sentences, learning rate 1.0, and roughly five epochs.Training continued until loss stopped decreasing on a validation set of 10,000 images and descriptions.Source: §5.2 · p. 7, §5.1 · p. 7
5.3 Results on the Gold RP10K and RP10M datasets
- Model A outperforms Word2Vec by 9.5% on Gold RP10K and 9.2% on RP10M; Model C also outperforms the pure-text RNN baseline.The paper reports that visual information helps when the model successfully fuses visual and text information.Source: §5.3 · pp. 7–8
- On Gold RP10K and RP10M, Model A outperforms its baseline without the weight-sharing strategy by 7.0% and 4.4%, respectively.Source: §5.3 · p. 8
- Model A performs best among Models A, B, and C; the authors suggest Model B struggles to pass its final-state loss gradient to embeddings in sentences longer than eight words.They attribute Model A’s advantage to soft supervision being more effective than direct supervision.Source: §5.3 · p. 8
- All Pinterest40M-trained models outperform skip-gram [25], which was trained on 300 billion words; the paper’s model trains on 3 billion words.Source: §5.3 · p. 8
6 Discussion
The paper concludes that visual information helps train word embeddings and that its models incorporate that information into learned embeddings. It identifies extensions to the evaluation data and phrase representations.
6 Discussion
- Experiments indicate that visual information helps train word embeddings and that the proposed models incorporate it into the learned embeddings.The paper introduces Pinterest40M, RP10M, and Gold RP10K as resources for training and evaluating the models.Source: §6 · p. 8
- Planned extensions include separating similarity from relatedness pairs in Gold RP10K, adding pair scores, and improving phrase representations.The authors say these additions would support more detailed evaluation and use of prior evaluation strategies.Source: §6 · p. 8
- Model A benefits from weight sharing, but applying the strategy to Models B and C yields performance similar to their non-sharing versions.Source: §6 · p. 8