Source-linked AI summary
The Role of Context Types and Dimensionality in Learning Word Embeddings
Oren Melamud, David McClosky, Siddharth Patwardhan, Mohit Bansal
TL;DR
The paper examines how context type and dimensionality affect skip-gram embeddings, an area without extensive evaluation across diverse contexts and NLP tasks. It evaluates multiple context types and combination methods, finding different patterns for intrinsic and extrinsic tasks and further gains from concatenation after dimensionality benefits are mostly exhausted.
Problem
There has not been an extensive evaluation of how diverse context types affect embeddings across a wide range of NLP tasks.
Source: §1 · p. 1Method
Results
Takeaways & Limitations
For extrinsic tasks, context type and dimensionality are typically task-specific choices, and combining complementary embeddings can further improve results.
Source: §7 · p. 9
Abstract
from arXiv · showhide
We provide the first extensive evaluation of how using different types of context to learn skip-gram word embeddings affects performance on a wide range of intrinsic and extrinsic NLP tasks. Our results suggest that while intrinsic tasks tend to exhibit a clear preference to particular types of contexts and higher dimensionality, more careful tuning is required for finding the optimal settings for most of the extrinsic tasks that we considered. Furthermore, for these extrinsic tasks, we find that once the benefit from increasing the embedding dimensionality is mostly exhausted, simple concatenation of word embeddings, learned with different context types, can yield further performance gains. As an additional contribution, we propose a new variant of the skip-gram model that learns word embeddings from weighted contexts of substitute words.
1 Introduction
The paper evaluates how context type and dimensionality affect skip-gram embeddings across intrinsic and extrinsic NLP tasks. It finds that extrinsic tasks require careful task-specific tuning, and that concatenation can help after dimensionality gains are mostly exhausted.
1 Introduction
- The study addresses a gap in broad comparisons of diverse context types, despite prior work showing benefits of syntactic contexts for functional similarity and selected downstream tasks.Prior examples include dependency parsing and lexical substitution.Source: §1 · p. 1
- Intrinsic evaluations predict human judgments of word relations, while extrinsic evaluations test downstream NLP tasks; prior work reports that their results do not correlate reliably.This motivates evaluating extrinsic tasks in addition to intrinsic benchmarks.Source: §1 · pp. 1–2
- The paper provides an extensive evaluation of context types and dimensionality across intrinsic tasks and four extrinsic NLP tasks.The extrinsic tasks are dependency parsing, named entity recognition, coreference resolution, and sentiment analysis.Source: §1 · p. 2
- For most considered extrinsic tasks, careful tuning of context type and dimensionality is needed, and concatenation can yield further gains once dimensionality benefits are mostly exhausted.The paper compares concatenation with dimensionality reduction using SVD or CCA.Source: §1 · p. 2
2 Word Embedding Context Types
The paper learns skip-gram embeddings from window, dependency, and substitute-word contexts using a shared corpus. Its qualitative comparison suggests that context choice changes whether nearest neighbors tend toward topical or functional similarity.
2.1 Learning Corpus
- All embedding types are learned from a shared corpus of about 10B words, with words below frequency 100 excluded and a vocabulary of about 500K.The corpus combines English Wikipedia 2015, the UMBC web corpus, and English Gigaword.Source: §2.1 · p. 2
2.2 Window-based Word Embeddings
- Window-based embeddings use word2vec skip-gram with negative sampling, where target-context pairs depend on window size and sampled negative contexts.The experiments use window sizes 1, 5, and 10, and vary embedding dimensionality.Source: §2.2 · p. 2
2.3 Dependency-based Word Embeddings
- Dependency embeddings pair targets with syntactic contexts built from modifiers and heads, including dependency relations and their inverses.The model uses parsed text and word2vecf; the objective uses dependency-based rather than window-based pairs.Source: §2.3 · p. 3
2.4 Substitute-based Word Embeddings
- Substitute vectors represent potential words that fit a target slot, weighted by their fit given the surrounding words.For “I love my job,” an example vector assigns weights to quit, love, hate, and lost.Source: §2.4 · p. 3
2.5 Qualitative Effect of Context Type
- Large-window embeddings yield both functionally and topically similar neighbors, while small-window and dependency embeddings generally show fewer topical neighbors.The qualitative inspection compares the most similar words for target words across embedding spaces.Source: §2.5 · p. 3
- Substitute-based embeddings show a stronger preference for functional similarity and tend to preserve verb tense.The paper inspects nearest neighbors using cosine similarity.Source: §2.5 · pp. 3–4
3 Word Embedding Combinations
The paper combines embeddings from different context types using concatenation or lower-dimensional projections. Concatenation preserves component vectors at increased dimensionality, while SVD and CCA reduce dimensionality in different ways.
3 Word Embedding Combinations
- Embedding combinations are treated as potentially more informative for some extrinsic tasks because context types produce embeddings with different properties.The paper contrasts lossless concatenation with lossy SVD and CCA combinations.Source: §3 · p. 4
3.1 Concatenation
- Concatenation joins each word’s vectors from two same-vocabulary embedding sets, adding their dimensions; experiments combine sets of equal dimensionality.The classifier may learn weights to choose suitable features for a task, such as dependency features for entity words and large-window features for surrounding words.Source: §3.1 · p. 4
3.2 Singular Value Decomposition
- SVD reduces the dimensionality of concatenated word embeddings, using a technique previously applied to compress sparse word representations.Source: §3.2 · p. 4
3.3 Canonical Correlation Analysis
- CCA treats embeddings learned from different context types as views of the same vocabulary and projects them to lower-dimensional spaces that maximize correlation.The paper tunes CCA hyperparameters using either SimLex-999 or WordSim-353-R, yielding projections tied to functional- or topical-similarity objectives.Source: §3.3 · p. 4
4 Evaluation
The evaluation pairs intrinsic semantic benchmarks with four downstream NLP tasks, assessing word embeddings through task-specific metrics and model inputs.
4.1 Intrinsic Benchmarks
- Intrinsic evaluation used cosine scores and Spearman correlation on WordSim-353, its functional/topical subsets, and SimLex-999; TOEFL used cosine-based accuracy on 80 synonym items.Source: §4.1 · p. 5
4.2 Extrinsic Benchmarks
- The downstream benchmarks covered dependency parsing, named entity recognition, coreference resolution, and sentiment; experiments mostly used development sets to explore performance trends.Source: §4.2 · p. 5
- The NNDEP parser was initialized with embeddings and evaluated on Penn Treebank WSJ sections 2–21 for training and 22 for development.Source: §4.2 · p. 5
- NER added embeddings as features to a regularized averaged perceptron, using CoNLL-2003 train/dev and MUC7 for out-of-domain testing.Source: §4.2 · p. 5
- Coreference added head-word and mention-average embeddings, with per-dimension antecedent, current-mention, and interaction features to Berkeley’s model.Source: §4.2 · p. 5
- Sentiment classification averaged word embeddings into sentence representations for L2-regularized logistic regression after neutral sentences were discarded.Source: §4.2 · pp. 5–6
5 Results
Across intrinsic and extrinsic tasks, preferred context types and dimensionalities vary. For extrinsic tasks, concatenation can yield further gains once increasing dimensionality offers little benefit.
5.1 Intrinsic Results for Context Types
- Intrinsic performance generally rises with dimensionality, nearing optimal around 300 dimensions, while context preferences differ by benchmark.Largest windows perform best on WordSim-353-R; DEP performs best on WordSim-353-S and SimLex-999; all contexts except SUB perform comparably on TOEFL.Source: §5.1 · p. 6
5.2 Extrinsic Results for Context Types
- For extrinsic tasks, optimal results may occur at 50 dimensions, and excessive dimensionality can degrade performance, making task-specific tuning important.DEP works best for PARSE, while context differences are difficult to distinguish in NER and COREF, where embeddings add only relatively small improvements over a competitive baseline.Source: §5.2 · pp. 6–7
- In out-of-domain NER MUC, every context type improves over the baseline by at least five points.The authors presume continuous embeddings are more robust to train-test differences, but leave detailed investigation of out-of-domain settings to future work.Source: Table 2 · p. 7, §5.2 · p. 8
5.3 Extrinsic Results for Combinations
- When dimensionality is near-optimal, concatenating embeddings can pay off; combinations of topical W10 with functional SUB, DEP, or W1 typically perform best.Concatenation does not always help: when increasing the dimensionality of single embeddings still yields significant improvement, doing so can be better.Source: §5.3 · p. 8
- SVD and CCA combinations degraded performance relative to single embeddings on all extrinsic tasks, unlike the concatenation approach.The authors suggest that preserving differences between context types may be beneficial.Source: §5.3 · p. 8
6 Related Work
Prior broad evaluations generally restricted context types, while studies of particular context types or embedding combinations addressed narrower settings. Earlier findings also show that intrinsic and extrinsic performance patterns can differ, with parsing as an exception.
6 Related Work
- Levy et al. (2015) and Lapesa and Evert (2014) evaluated intrinsic tasks with word-window contexts, while Lai et al. (2015) included extrinsic tasks but used the default window size of 5.The paper positions its diverse-context evaluation as distinct from these earlier broad evaluations.Source: §6 · pp. 8–9
- Schnabel et al. (2015) and Tsvetkov et al. (2015) reported low intrinsic–extrinsic correlation without evaluating different context types; the paper finds differing performance patterns except in parsing.These results are consistent with the paper’s findings.Source: §6 · p. 9
- Bansal et al. (2014) found functional dependency-based and small-window embeddings yielded greater parsing improvements than topical large-window embeddings, consistent with this paper’s findings.Source: §6 · p. 9
- Yin and Schütze (2015) and Goikoetxea et al. (2016) combined embeddings but evaluated mostly intrinsic tasks and did not consider diverse context types.Manually constructed resources such as WordNet were another potentially complementary approach (Faruqui et al., 2014; Kiela et al., 2015).Source: §6 · p. 9
7 Conclusions
The conclusions emphasize that embedding settings behave differently across intrinsic and extrinsic tasks. Extrinsic performance depends on task-specific choices of context type and dimensionality, and complementary embeddings can add gains.
7 Conclusions
- Skip-gram embeddings were evaluated across multiple intrinsic and extrinsic tasks while varying context type and dimensionality.Source: §7 · p. 9
- For extrinsic tasks, optimal context type and dimensionality are critical and typically task-specific, although standard hyperparameter practices typically work well for intrinsic tasks.Source: §7 · p. 9
- Combining complementary embeddings learned with different context types and suitable dimensionality can often yield further improvements.Source: §7 · p. 9