PaperlayerView exact revision on arXiv

Source-linked AI summary

How to Generate a Good Word Embedding?

Siwei Lai, Kang Liu, Liheng Xu, Jun Zhao

arXiv:1507.05523v1cs.CL

TL;DR

The paper addresses limited fair comparisons of neural word embedding algorithms and how to design embeddings for specific tasks. It systematizes model, corpus, and parameter choices, evaluates them across multiple tasks, and derives practical training guidelines. Corpus domain matters more than size, simpler models usually suffice, and task development-set performance is preferable to embedding validation loss for early stopping.

  • Problem

    Few studies fairly compare word embedding algorithms, leaving it unclear how to design an effective embedding for a specific task.

  • Method

    The paper systematizes neural word embedding algorithms by target-context relationships and context representation, then evaluates models across multiple tasks while analyzing corpus and training-parameter choices.

  • Results

    Corpus domain is more important than corpus size; faster models are sufficient in most cases; and development-set task performance is a better early-stopping metric than embedding validation loss.

  • Takeaways & Limitations

    Choose an in-domain corpus before enlarging it, use complex models mainly with sufficiently large corpora, and stop training according to the desired task's development set.

  • Takeaways & Limitations

    Some corpus-size exceptions may be attributable to instability in the evaluation metrics.

Abstract

from arXiv · show

We analyze three critical components of word embedding training: the model, the corpus, and the training parameters. We systematize existing neural-network-based word embedding algorithms and compare them using the same corpus. We evaluate each word embedding in three ways: analyzing its semantic properties, using it as a feature for supervised tasks and using it to initialize neural networks. We also provide several simple guidelines for training word embeddings. First, we discover that corpus domain is more important than corpus size. We recommend choosing a corpus in a suitable domain for the desired task, after that, using a larger corpus yields better results. Second, we find that faster models provide sufficient performance in most cases, and more complex models can be used if the training corpus is sufficiently large. Third, the early stopping metric for iterating should rely on the development set of the desired task rather than the validation loss of training embedding.

1. INTRODUCTION

The paper addresses the lack of fair, task-oriented comparisons for word embedding algorithms by systematically analyzing models, corpora, and training parameters. Across multiple evaluation types, it derives practical guidelines for choosing models, corpora, dimensions, and stopping criteria.

  • The study targets the limited evidence on designing effective word embedding algorithms for specific tasks.
  • It analyzes model construction, corpus size and domain, and training parameters, including iterations and embedding dimensionality.
  • The authors systematize neural word embedding algorithms by target-context relationships and context representations, then evaluate them across multiple tasks.
  • More complex models require larger training corpora to outperform simpler models, which are sufficient in most cases.
  • Corpus domain is more important than corpus size, although larger in-domain corpora generally improve embedding quality.
  • Task development-set performance is preferred over embedding validation loss for early stopping, while dimensionality 50 is sufficient for feature and initialization tasks.

2. MODELS

The paper denotes the embedding of word w by e(w) when describing and comparing its word embedding models.

  • The embedding of word w is denoted e(w) throughout the model descriptions and experiments.

2.1 Model Overview

The model overview covers neural architectures that either predict a target word from context or score a target word together with its context. It contrasts richer context representations with simpler, faster alternatives and also notes count-based models.

  • 2.1.1 NNLM: NNLM learns word embeddings and a language model by predicting the next word from several previous words.
  • 2.1.1 NNLM: NNLM uses concatenated previous-word embeddings as input to a feed-forward neural network with one hidden layer and a softmax output.
  • 2.1.1 NNLM: LBL is similar to NNLM but removes the nonlinear tanh activation, while HLBL and ivLBL accelerate related models computationally.
  • 2.1.2 C&W: C&W trains embeddings by scoring the concatenation of a central target word and its context, using corrupted sequences with randomly replaced targets.
  • 2.1.3 CBOW and Skip-gram: CBOW averages context-word embeddings and Skip-gram uses one context word, both simplifying computation while neglecting most word-order information.
  • 2.1.4 Order and 2.1.5 GloVe: The Order model preserves word order through concatenation while removing the hidden layer, and GloVe represents a count-based word-context-matrix approach.

2.2 Model Analysis

The models differ in how they relate target words to context and how they represent that context. Their representations range from selecting or averaging context embeddings to preserving order or learning compositionality with hidden layers.

  • Existing neural embedding models either use context to predict the target word or model the target-context combination directly.
  • Skip-gram selects one context-word embedding, whereas CBOW averages the context embeddings.
  • Order concatenates context-word embeddings to maintain word order, while LBL and NNLM add hidden layers that encode context compositionality.

3. TASKS

The evaluation covers eight tasks in three categories: semantic properties, supervised NLP features, and neural-network initialization. These tasks use established datasets and task-specific metrics to assess embeddings.

  • Evaluation task types: Eight tasks are grouped into semantic evaluation, supervised NLP feature use, and neural-network initialization.The paper argues these three types cover current word-embedding applications.
  • Semantic properties: WordSim353 measures Pearson correlation between embedding cosine distances and human ratings for 353 word pairs.
  • Semantic properties: TOEFL evaluates synonym selection by nearest-neighbor cosine distance across 80 multiple-choice questions.Each question has four candidate answers, and performance is measured by accuracy.
  • Semantic properties: The analogy task evaluates semantic and syntactic relations using approximately 9K semantic and 10.5K syntactic questions, scored by accuracy.Answers are selected by nearest-neighbor search after vector arithmetic.
  • Supervised NLP tasks: Supervised feature evaluation uses text classification with averaged embeddings and named entity recognition with embeddings added to a near-state-of-the-art system.The classification task uses IMDB, while NER uses CoNLL03 and reports F1.
  • Neural-network initialization: Initialization evaluation uses embeddings to initialize neural networks for CNN sentiment classification and POS tagging.CNN experiments use Stanford Sentiment Treebank accuracy, while POS tagging uses Wall Street Journal accuracy.

4. EXPERIMENTS AND RESULTS

The experiments compare selected embedding models trained on different corpora and parameter settings across the paper’s eight evaluation tasks.

  • The experiments compare embedding models across different corpora and parameter settings using eight tasks.Table 3 summarizes the selected models, corpora, and hyperparameters.

4.1 Performance Gain Ratio

The Performance Gain Ratio addresses cross-task comparison by normalizing performance relative to random and best embeddings. It is designed to make heterogeneous task results easier to compare.

  • Different task metrics have different ranges and variances, making raw performance comparisons across tasks difficult.For example, POS scores typically exceed 96%, while WordSim353 and text classification occupy different ranges and spreads.
  • The Performance Gain Ratio replaces raw metrics such as accuracy and F1 to compare embeddings across tasks.
  • PGR compares each embedding’s gain over a random embedding with the gain of the best embedding in the same setting.The random embedding matches the evaluated embedding’s dimensionality, with dimensions uniformly sampled from −1 to 1.
  • PGR = 100% denotes the best result, PGR = 0 indicates no more useful information than random, and PGR < 0 indicates detrimental performance.
  • PGR normalizes performance across tasks to a common scale, simplifying multi-task embedding analysis.The paper uses PGR when comparing embeddings across tasks.

4.2 Model and Implementation

The comparison evaluates word-embedding models across semantic, feature-based, and neural-network-initialization tasks using common implementations and corpora. Simpler models are generally sufficient, while larger corpora can make more complex models advantageous; target-word prediction also outperforms C&W on semantic tasks.

  • Comparison setup: The experiments compare different context representations and target-context relationships across eight evaluation tasks.All models use the same implementation framework, with task performance evaluated after convergence or overfitting.
  • Target-context relationship: In semantic tasks, target-word-predicting models outperform C&W, which jointly places the target word and context in the input layer.The C&W model performs lower on semantic-property tasks and almost entirely lacks linear semantic subtraction in analogy tasks.
  • Target-context relationship: C&W produces less paradigmatic nearest neighbors than target-word-predicting models, reflecting a weaker capture of relations among words that share predictive contexts.Examples include time-of-day neighbors for “Monday” and co-occurring words for “commonly,” rather than substitute-like alternatives.
  • Model complexity and corpus size: Skip-gram performs best with 10M- or 100M-token corpora, whereas more complex models win more often on larger corpora.For corpora larger than those tested, LBL and NNLM may be preferable.
  • Model complexity and task type: For feature-based and neural-network-initialization tasks, model choice does not significantly affect results, so simpler models are typically sufficient.The margins between simple and complex models are relatively small in these real-task settings.
  • Model complexity and corpus size: In smaller corpora, simpler models such as Skip-gram perform better, while CBOW and Order are typically superior with larger corpora.The paper concludes that Skip-gram, CBOW, and Order are generally adequate for real tasks.

4.3 The Effect of the Training Corpus

The corpus experiments show that domain suitability matters more than corpus size for word-embedding quality. In-domain data can substantially improve task performance, while larger corpora help mainly when their domains match.

  • Corpus size: Using a larger corpus generally improves embeddings when compared corpora share the same domain.The larger corpus is superior in almost all same-domain comparisons, with exceptions attributed to evaluation-metric instability.
  • Corpus domain: The 100M-token Wiki subset outperforms the 1.2B-token NYT corpus on semantic tasks such as semantic analogies and similarity.The paper attributes this pattern to Wikipedia’s more comprehensive knowledge, which may benefit semantic tasks.
  • Corpus domain: The small IMDB corpus benefits avg and cnn tasks but performs very poorly on ner and pos tasks.Because IMDB matches the avg and cnn data source, its in-domain advantage is especially strong; its informal style is detrimental to pos.
  • Corpus domain: Nearest-neighbor examples show that corpus domain changes word associations, including “season” referring to television episodes in IMDB and sports competition in W&N.The paper uses these shifts to illustrate how in-domain corpora produce task-suitable embeddings.
  • Corpus domain: Corpus domain has a greater effect than corpus size: suitable in-domain data improves performance, whereas unsuitable domains may reduce it.For specific tasks, a pure in-domain corpus outperforms a mixed-domain corpus; within the same domain, larger corpora perform better.

4.4 The Choice of the Training Parameters

The section examines iteration and dimensionality choices for word embeddings. It finds that task performance is better aligned with task-based stopping criteria than validation loss, while dimensionality needs depend on the evaluation use.

  • Number of Iterations: Validation loss can diverge from real-task performance, peaking too early on 100M-token data while some task scores continue increasing.On 1B-token data, NER and POS performance instead decline after several iterations despite no apparent validation overfitting.
  • Number of Iterations: Stopping at the peak of a simple task won in 117 cases, compared with 89 cases when stopping at peak validation loss.The comparison covers 168 combinations of tasks, models, and corpus subsets, with a win defined as reaching 95% of peak task performance.
  • Number of Iterations: For task-specific training, use the development-set performance of the desired task to choose when to stop iterating.When evaluation is too time consuming, another task such as TFL can provide an approximation; CNN and POS checks take dozens of minutes, whereas TFL takes seconds.
  • Number of Iterations: Multiple iterations are necessary because performance increases substantially beyond one pass across tasks and corpora.The finding suggests that single-pass training, as in an early version of word2vec, can lose performance.
  • Dimensionality: Larger dimensions improve semantic-property tasks, whereas dimensionality 50 is typically sufficient when embeddings serve as features or initialization.The exception is the C&W model for semantic-property tasks.

5. RELATED WORK

Related work includes comparisons of embedding models, count-based and predictive representations, and the effects of corpus size and domain. The cited studies motivate broader, fairer comparisons across models, tasks, and corpus conditions.

  • Model comparisons: Turian et al. compared HLBL and C&W embeddings on NER and chunking using a 63M-token corpus, finding similar task improvements.Their comparison used a relatively small corpus.
  • Model comparisons: Baroni et al. compared count models with predictive neural embeddings, using CBOW as the predictive model in experiments across semantic evaluation tasks.The supplied passage does not report the comparison outcome.
  • Corpus effects: Prior work has often examined corpus size, while other studies have also considered corpus domain and its relationship to task performance.Reported examples include larger CBOW corpora improving analogy performance and in-domain corpora improving biomedical NER over newswire corpora.

6. CONCLUSION

The paper fairly compares word embedding models across tasks while analyzing model, corpus, and training-parameter choices. It does not identify one universally best setting, but provides practical training guidelines.

  • Conclusion: The study analyzes three critical components of word embedding training: the model, the corpus, and the training parameters.It compares various models across various tasks under a common evaluation framework.
  • Conclusion: No specific training setting achieves the best performance on all tasks, and such a setting may not exist.The paper therefore presents guidelines rather than a single universally optimal configuration.
Loading 1507.05523v1…