PaperlayerView exact revision on arXiv

Source-linked AI summary

Do Multi-Sense Embeddings Improve Natural Language Understanding?

Jiwei Li, Dan Jurafsky

arXiv:1506.01070v3cs.CL

TL;DR

The paper asks whether multi-sense embeddings improve real NLU tasks beyond artificial similarity judgments. It introduces a Chinese Restaurant Process model and a pipeline for applying sense-specific embeddings across tasks. Gains appear in some tasks but not others, and increasing dimensionality can often match them.

  • Problem

    It is unclear whether multi-sense embeddings can be incorporated into and benefit real-world NLU tasks.

  • Method

    The paper proposes a Chinese Restaurant Process multi-sense embedding model and a pipeline for applying it to multiple NLP tasks.

  • Results

    Multi-sense embeddings improve POS tagging, semantic relation identification, and semantic relatedness, but not NER or sentiment analysis.

  • Takeaways & Limitations

    Testing embeddings in real NLP applications reveals task-dependent benefits that artificial similarity evaluations alone do not establish.

  • Takeaways & Limitations

    The conclusions are based on the proposed pipelined system, and other multi-sense systems may find stronger effects.

Abstract

from arXiv · show

Learning a distinct representation for each sense of an ambiguous word could lead to more powerful and fine-grained models of vector-space representations. Yet while `multi-sense' methods have been proposed and tested on artificial word-similarity tasks, we don't know if they improve real natural language understanding tasks. In this paper we introduce a multi-sense embedding model based on Chinese Restaurant Processes that achieves state of the art performance on matching human word similarity judgments, and propose a pipelined architecture for incorporating multi-sense embeddings into language understanding. We then test the performance of our model on part-of-speech tagging, named entity recognition, sentiment analysis, semantic relation identification and semantic relatedness, controlling for embedding dimensionality. We find that multi-sense embeddings do improve performance on some tasks (part-of-speech tagging, semantic relation identification, semantic relatedness) but not on others (named entity recognition, various forms of sentiment analysis). We discuss how these differences may be caused by the different role of word sense information in each of the tasks. The results highlight the importance of testing embedding models in real applications.

1 Introduction

The paper asks whether multi-sense embeddings can benefit real NLU tasks, not only artificial similarity judgments. It introduces a pipelined architecture and finds task-dependent gains, with dimensionality and model sophistication affecting comparisons.

  • Motivation and approach: Multi-sense embeddings require sense-specific learning, sense induction for tokens, and representation acquisition for phrases or sentences.The resulting representations are passed to machine-learning classifiers.
  • Motivation and approach: The proposed pipeline evaluates multi-sense embeddings on POS tagging, NER, sentiment analysis, semantic relation identification, and semantic relatedness.
  • Findings: Multi-sense embeddings improve some tasks, including POS tagging, semantic relation identification, and semantic similarity, but not NER or sentiment analysis.The paper offers suggested explanations for these differences.
  • Findings: Some multi-sense improvements disappear with more sophisticated neural models such as LSTMs, which can filter irrelevant information more flexibly.
  • Evaluation considerations: Matching embedding dimensionality is essential because combining global and sense-specific vectors can double dimensionality relative to a 50d baseline.The paper reports that simply increasing dimensionality is often the most straightforward route to better performance.

2 Related Work

Standard embeddings assign one vector to each word, while multi-sense methods associate words with multiple sense-specific vectors. Earlier work largely evaluated these models on artificial similarity judgments rather than general NLP tasks.

  • Single-vector embeddings: Standard neural embedding models represent each word with a single dense vector learned from neighboring-word prediction or co-occurrence structure.
  • Multi-sense embeddings: Multi-sense methods augment embedding models by associating each word with several vectors and disambiguating senses using local co-occurrence.For example, fruit and technology contexts distinguish different senses of “apple.”
  • Prior approaches: Prior approaches used context clustering, fixed or variable numbers of senses, and sense/topic-specific embeddings.
  • Evaluation gap: Earlier multi-sense research was mostly evaluated on the relatively artificial task of matching human word similarity judgments.

3 Learning Sense-Specific Embeddings

The proposed model learns multiple sense embeddings with a Chinese Restaurant Process that balances contextual relatedness and sense popularity. It jointly performs sense clustering and embedding learning within a distributed language model.

  • Chinese Restaurant Process: Chinese Restaurant Processes choose an existing sense or create a new one using semantic relatedness and the popularity of existing senses.The CRP combines local context and global document information.
  • Chinese Restaurant Process: The CRP probability for an existing table depends on its customer count and word probability, while γ controls preference for a new table.
  • Chinese Restaurant Process: CRPs provide a “rich get richer” effect while avoiding the fixed threshold strategy for creating new clusters.
  • Model representation: The model assigns each word type a global embedding plus a set of distinct sense-specific embeddings, with the number of senses discovered per word.
  • Training pipeline: The distributed language model first predicts a token from context and then uses CRP to assign its occurrence to a discovered sense or construct a new one.
  • Joint learning: During training, word-sense clustering and embedding learning are joint: current-token prediction uses neighboring global and sense-specific vectors, and updates both.

4 Obtaining Word Representations for NLU tasks

For downstream NLU, the paper keeps embeddings fixed while inferring token senses and building task-level representations. It compares greedy and expectation-based sense representations, alongside standard embeddings, on contextual similarity evaluation.

  • Sense inference: At inference, global and sense-specific embeddings are fixed while the system assigns sense labels to tokens in a sentence or document.
  • Sense inference: Exact global sense labeling can be expensive, so the paper uses greedy local assignment and expectation vectors as simplified heuristics.
  • SCWS evaluation: 66.4 and 67.0 are the SCWS correlations for SG+Greedy and SG+Expect on the 1.1B-word Wikipedia dataset, versus 64.6 for SkipGram.
  • SCWS evaluation: 69.1 and 69.7 are the SCWS correlations for SG+Greedy and SG+Expect on the 120B-word dataset, versus 66.4 for SkipGram.

5 Word Similarity Evaluation

The paper evaluates multi-sense embeddings on contextual word similarity, using large corpora and context-based inference to compare Greedy and Expectation strategies. Multi-sense embeddings improve contextual similarity, with Expectation outperforming Greedy and the one-vector baseline.

  • Evaluation setup: The evaluation uses Stanford’s Contextual Word Similarities task, where human judgments concern word pairs presented in context.This addresses the context-free limitation of earlier similarity datasets.
  • Evaluation setup: Models are trained on Wikipedia or a much larger combined corpus, then infer token vectors with Greedy or Expectation strategies and compare words using cosine similarity.The combined corpus contains 120 billion tokens, while the Wikipedia corpus contains 1.1 billion.
  • Results: Multi-sense embeddings outperform the single-vector skip-gram model on the context-dependent SCWS task.Both SG+Greedy and SG+Expect perform better than SG.
  • Results: +2.4 performance boost is obtained by SG+Expect over the one-word-one-vector strategy on the 1.1 billion-token Wikipedia dataset.The corresponding boost on the Common Crawl dataset is +3.2.
  • Results: Expectation inference outperforms Greedy inference because neighboring words alone do not provide sufficient information for word-sense disambiguation.Expectation incorporates global-level information when choosing senses.
  • Results: Contextual nearest-neighbor examples show that the model disambiguates senses of words such as bank, rock, and apple using local context.The model also handles polysemy involving less distinct senses.

6 Experiments on NLP Tasks

The paper tests multi-sense embeddings across several NLP tasks while controlling for dimensionality. They help some tasks, including POS tagging, semantic relation identification, and semantic relatedness, but not NER or sentiment analysis; some gains disappear with more sophisticated neural models.

  • Experimental design: The experiments compare standard, Greedy, Expectation, concatenated global-plus-sense, doubled-dimension, and 300d embeddings across multiple NLP tasks.The doubled 100d one-vector model is the appropriate baseline for the 100d concatenated representation.
  • Experimental design: The study uses neural architectures for tagging and sentence representation, including recurrent or recursive models and LSTMs for sentence-level relatedness.Word embeddings are used as inputs to task-specific models and are kept fixed in the Pang sentiment experiment.
  • Dimensionality control: Comparisons with the matched 100d baseline make the apparent universal advantage of concatenated representations more nuanced.The initial comparison against the 50d baseline is unfair because concatenation doubles vector dimensionality.
  • Overall results: Expectation models often outperform simple baselines at both 50d and 100d, but only some differences are statistically significant.Significance testing compares 50d standard with 50d Expectation and 100d standard with 100d Expectation.
  • Task-specific results: Multi-sense embeddings provide little significant benefit for sentiment analysis because common sentiment words often retain similar sentiment across senses.The discussion also questions their usefulness for capturing local composition such as negation.
  • Task-specific results: Multi-sense embeddings improve POS tagging but not NER, consistent with word senses being more closely related to POS than to the dominant non-entity tag.NER comparisons report p-values of 0.223 for 50d and 0.310 for 100d, while POS reports 0.033 and 0.031.
  • Model capacity: The advantages of multi-sense embeddings disappear for semantic relatedness when using more sophisticated LSTM models.This suggests that model architecture can reduce the observed difference between single-vector and multi-sense representations.

7 Conclusion

The paper introduces a Chinese Restaurant Process multi-sense embedding model and a pipeline for testing multi-sense representations in NLP applications. Results show task-dependent gains, while dimensionality increases can sometimes provide the same improvements.

  • 7 Conclusion: The paper proposes a Chinese Restaurant Process model that learns multiple embeddings for each word type and improves performance on word similarity tasks.The model assigns tokens to existing or new sense clusters using contextual relatedness and sense popularity.
  • 7 Conclusion: Increasing baseline embedding dimensionality sometimes achieves the same performance improvements as using multi-sense embeddings.The paper identifies dimensionality increases as the most straightforward route to better performance on these tasks.
  • 7 Conclusion: Multi-sense embeddings improve some tasks, including part-of-speech tagging and semantic relation identification, but not all NLP tasks.The reported benefits are consistent for selected tasks, while other task outcomes are less favorable.
  • 7 Conclusion: The conclusions are limited to the proposed pipeline, although the authors emphasize evaluating embeddings in real NLP applications beyond human-matching tasks.Other sense-learning or pipeline systems may produce stronger multi-sense effects.

8 Appendix

The appendix describes recurrent, recursive, and LSTM architectures for composing token or child representations into phrase- and sentence-level vectors. These models differ in whether they process sequences, parse-tree neighborhoods, or gated memory states.

  • Recurrent Networks: Recurrent networks combine each token vector with the previous hidden vector to build a sequence representation step by step.The final hidden vector represents the whole sequence.
  • Recurrent Networks: A recurrent model uses compositional matrices W and V to compute the current hidden representation from the token and prior hidden state.For a sequence of length N_s, h_Ns represents the whole sequence S.
  • Recursive Networks: Recursive networks compose neighboring words according to parse-tree order rather than sequence order.Each parent representation is computed from its immediate children in a bottom-up traversal until the root.
  • Long Short Term Memory (LSTM): LSTMs process sequences using input, memory, and output gates while combining each input representation with the preceding hidden state.The formulation distinguishes token or text-unit vectors e_t from the LSTM-computed states h_t.

http://arxiv.org/ps/1506.01070v3

The appendix references four figures by filename, but the supplied passages provide no captions, axes, encodings, or stated comparisons for them.

  • Appendix: Figures 2–5 are referenced as PNG files without accompanying descriptions of their visual encodings or results.No supported comparison can be extracted from these references alone.
Loading 1506.01070v3…