Source-linked AI summary
Combining Language and Vision with a Multimodal Skip-gram Model
Angeliki Lazaridou, Nghia The Pham, Marco Baroni
TL;DR
Text-only semantic models lack grounding in extra-linguistic modalities, while existing multimodal approaches have limited integration and coverage. MMSKIP-GRAM extends skip-gram with jointly learned linguistic and visual objectives for selected words, propagating visual information across the vocabulary. The resulting representations perform well across semantic and vision tasks, including zero-shot image labeling and retrieval, and show visual structure for some abstract words.
Problem
Text-only semantic models lack grounding in extra-linguistic modalities, while existing multimodal models separately construct modalities, assume broad coverage, and lack cross-modal generalization.
Method
MMSKIP-GRAM extends skip-gram by jointly learning to predict linguistic contexts and visual representations presented with contexts for a subset of target words.
Results
MMSKIP-GRAM representations perform well on semantic benchmarks and support zero-shot image labeling and retrieval, while indirect visual evidence affects abstract-word representations.
Takeaways & Limitations
The models propagate visual information beyond directly observed words and may support applications involving visual knowledge, abstract concepts, and grounded language.
Takeaways & Limitations
Evaluation includes cases where model-predicted visual neighbors have no obvious relation to abstract words and are strongly disfavored by subjects.
Abstract
from arXiv · showhide
We extend the SKIP-GRAM model of Mikolov et al. (2013a) by taking visual information into account. Like SKIP-GRAM, our multimodal models (MMSKIP-GRAM) build vector-based word representations by learning to predict linguistic contexts in text corpora. However, for a restricted set of words, the models are also exposed to visual representations of the objects they denote (extracted from natural images), and must predict linguistic and visual features jointly. The MMSKIP-GRAM models achieve good performance on a variety of semantic benchmarks. Moreover, since they propagate visual information to all words, we use them to improve image labeling and retrieval in the zero-shot setup, where the test concepts are never seen during model training. Finally, the MMSKIP-GRAM models discover intriguing visual properties of abstract words, paving the way to realistic implementations of embodied theories of meaning.
1 Introduction
Purely textual distributional models derive useful semantic representations but lack grounding in extra-linguistic modalities. MMSKIP-GRAM jointly learns from corpus contexts and visual evidence, propagating visual information beyond directly observed words.
- Text-only distributional semantic models represent meaning from corpus co-occurrence patterns but lack grounding in extra-linguistic modalities.
- Existing multimodal models often build linguistic and visual representations separately, assume coverage for all words, and lack cross-modal generalization.
- MMSKIP-GRAM extends skip-gram by presenting visual evidence with corpus contexts for a subset of target words.
- A joint linguistic-visual objective propagates visual information to words without direct visual evidence during training.
- The models perform well on semantic benchmarks, support zero-shot image labeling and retrieval, and reveal visual properties of abstract words.
2 Related Work
Earlier multimodal models commonly merge separately constructed text and image representations or use limited perceptual and captioned-image data. This paper instead develops general-purpose multimodal word representations for broad language-vision applications.
- Bruni et al. independently construct text and image vectors, then fuse their concatenation with Singular Value Decomposition.
- Silberer and Lapata use high-level visual attributes and a multimodal fusion strategy that empirically outperforms the simpler approach.
- Howell et al. propagate grounding from early concrete words incrementally, but rely on subject-generated features and small-scale word-learning simulations.
- Hill and Korhonen integrate subject-generated perceptual features and image-annotation text into skip-gram representations.
- Zero-shot learning maps images of unseen objects into linguistic space for labeling, rather than combining visual and linguistic information in a common space.
- Caption-based language-vision systems use limited captioned-image collections, whereas this work automatically enriches a very large corpus with images.
3 Multimodal Skip-gram Architecture
The architecture begins with skip-gram context prediction and adds a visual objective for words with visual evidence. MMSKIP-GRAM-A directly aligns modalities, while MMSKIP-GRAM-B learns a regularized cross-modal mapping.
- 3.1 Skip-gram Model: Skip-gram learns word representations by predicting context words surrounding each target in a text corpus.
- 3.1 Skip-gram Model: Hierarchical softmax reduces the softmax computation from O(|W|) to O(log |W|).
- 3.2 Injecting visual knowledge: In situated learning, some target words receive visual representations alongside their corpus contexts.
- 3.2 Injecting visual knowledge: The multimodal objective sums linguistic and visual terms, with the visual term set to zero when no visual information is available.
- 3.3 Multi-modal Skip-gram Model A: MMSKIP-GRAM-A uses a max-margin objective with negative visual samples to make each word representation more similar to its own visual vector than to others.
- 3.4 Multi-modal Skip-gram Model B: MMSKIP-GRAM-B inserts a learned cross-modal mapping matrix, removing the equal-dimensionality constraint and adding L2 regularization.
4 Experimental Setup
The experiments train on a large Wikipedia corpus and add ImageNet-derived visual information for a restricted, concrete vocabulary. Both baseline and multimodal models use 300-dimensional hidden representations with specified optimization settings.
- The training corpus is a Wikipedia 2009 dump containing approximately 800M tokens.
- Visual information is added for 5,100 ImageNet words meeting frequency and concreteness criteria, covering about 5% of corpus tokens.
- Each word’s visual representation averages 100 ImageNet picture vectors extracted as 4096-dimensional FC7 activations from a pretrained convolutional network.
- All models use a 300-dimensional hidden layer, while MMSKIP-GRAM-A retains the first 300 visual dimensions for comparability.
- Training uses hierarchical softmax, frequency subsampling t = 0.001, window size c = 5, and tuned multimodal margin and sampling parameters.
5 Experiments
The experiments show that multimodal skip-gram representations perform strongly across semantic similarity benchmarks, support zero-shot image tasks, and propagate visual structure to abstract-word representations. Qualitative and intrinsic analyses further suggest that visual grounding changes semantic neighborhoods and captures greater representational diversity for abstract concepts.
- Approximating human judgments: MMSKIP-GRAM models perform at or near the top across visual-coverage semantic similarity tasks and consistently outperform SKIP-GRAM and SVD on full datasets.They approach MEN state-of-the-art correlation 0.8, while MMSKIP-GRAM-B is near the reported Simlex-999 state of the art at 0.41.
- Approximating human judgments: Multimodal neighborhoods shift from contextual associations toward visually or taxonomically related concepts, including pizzas for donuts and birds of prey for owls.These changes also affect words without direct visual training evidence through visual propagation.
- Zero-shot image labeling and retrieval: Both multimodal models outperform SKIP-GRAM in zero-shot image labeling, while MMSKIP-GRAM-A performs especially well for higher-ranked image retrieval results.MMSKIP-GRAM-B achieves the best image-retrieval performance overall among the discussed models.
- Abstract words: Nearest visual neighbors are meaningfully related to about half of the evaluated words, including roughly one fourth of abstract terms, although concrete words fare better.Direct visual exposure during training strongly affects outcomes, and only two abstract words received matching images.
- Abstract words: Human judgments strongly favor some abstract-word visual neighbors, such as those associated with freedom, god, and theory, but not all examples show an obvious relation.
- Abstract words: Abstract-word representations show higher visual diversity than concrete-word representations, and MMSKIP-GRAM-A plus mapped MMSKIP-GRAM-B* entropy correlate highly with concreteness.The result supports the hypothesis that grounded abstract concepts encode varied and composite situations.
6 Conclusion
The paper introduces two multimodal extensions of SKIP-GRAM that integrate visual and linguistic information. Their representations perform well across semantic and vision tasks and support applications involving visual knowledge.
- MMSKIP-GRAM-A directly optimizes similarity between words and visual representations, while MMSKIP-GRAM-B adds a mediating cross-modal mapping layer.
- Integrating and propagating visual information produces word representations that perform well in both semantic and vision tasks.
- The representations can serve systems that benefit from prior visual knowledge, including caption generation, and may support metaphor detection or retrieving and generating images of abstract concepts.
- Their incremental training makes the models suitable for cognitive simulations of grounded language acquisition.