PaperlayerView exact revision on arXiv

Source-linked AI summary

Do Multi-Sense Embeddings Improve Natural Language Understanding?

Jiwei Li, Dan Jurafsky

arXiv:1506.01070v3cs.CL

TL;DR

It was unclear whether multi-sense embeddings could be incorporated into and improve real-world language understanding tasks. The paper introduces a Chinese Restaurant Process-based embedding model and a pipeline for evaluating it, finding benefits on some tasks but not others, with results limited to that pipeline.

  • Problem

    It was unclear how multi-sense embeddings could be incorporated into and benefit real-world natural language understanding tasks.

    Source: §1 · p. 2

  • Method

    The paper proposes a Chinese Restaurant Process-based multi-sense embedding model and a pipelined architecture applied to several NLP tasks.

    Source: §1 · p. 2, §7 · p. 8

  • Results

    Multi-sense embeddings improved part-of-speech tagging, semantic relation identification, and semantic relatedness, while sentiment-analysis improvement was nonsignificant; increased dimensionality sometimes matched their gains.

    Source: §6.2 · p. 8, §7 · p. 9

  • Takeaways & Limitations

    The results support evaluating embedding models in real NLP applications, beyond simple human-matching tasks.

    Source: §7 · p. 9

    The conclusions are based on the proposed pipeline, and other multi-sense embedding or pipeline systems may find stronger effects.

    Source: §7 · p. 9

Abstract

from arXiv · show

Learning a distinct representation for each sense of an ambiguous word could lead to more powerful and fine-grained models of vector-space representations. Yet while `multi-sense' methods have been proposed and tested on artificial word-similarity tasks, we don't know if they improve real natural language understanding tasks. In this paper we introduce a multi-sense embedding model based on Chinese Restaurant Processes that achieves state of the art performance on matching human word similarity judgments, and propose a pipelined architecture for incorporating multi-sense embeddings into language understanding. We then test the performance of our model on part-of-speech tagging, named entity recognition, sentiment analysis, semantic relation identification and semantic relatedness, controlling for embedding dimensionality. We find that multi-sense embeddings do improve performance on some tasks (part-of-speech tagging, semantic relation identification, semantic relatedness) but not on others (named entity recognition, various forms of sentiment analysis). We discuss how these differences may be caused by the different role of word sense information in each of the tasks. The results highlight the importance of testing embedding models in real applications.

1 Introduction

The introduction proposes a pipeline for applying multi-sense embeddings to NLP tasks and reports that their benefits vary by task, model, and embedding dimensionality.

1 Introduction

  • Conventional embeddings assign one vector per word type, forcing ambiguous words to combine senses, while how multi-sense embeddings benefit real NLU tasks remains unclear.Source: §1 · pp. 1–2
  • The proposed pipeline learns sense-specific embeddings, infers token senses in a text unit, then builds phrase or sentence representations for classifiers.Sense-specific embeddings may be learned unsupervised or with external resources such as WordNet.Source: §1 · p. 1
  • Multi-sense embeddings improve word and sentence similarity, semantic relation identification, and part-of-speech tagging, but not named entity extraction or sentiment analysis.The proposed evaluation covers part-of-speech tagging, named entity recognition, sentiment analysis, semantic relation identification, and semantic relatedness.Source: §1 · p. 2
  • Some multi-sense improvements are no longer visible with LSTMs, which can filter informational chaff from the wheat.Source: §1 · p. 2
  • At equal embedding dimensionality, simply increasing dimensionality is the most straightforward way to improve performance on these tasks.The paper emphasizes that embedding comparisons should control for dimensionality.Source: §1 · p. 2

2 Related Work

Standard neural embedding models give each word one vector, while multi-sense work assigns sense-specific vectors using contextual co-occurrence. Prior approaches differ in how they derive senses, and this literature has mostly been assessed on human word-similarity judgments.

2 Related Work

  • Neural embedding frameworks learn dense token vectors by predicting neighboring words or decomposing co-occurrence matrices, while standard models give each word one vector.Examples cited include (3; Collobert and Weston, 2008; Mnih and Hinton, 2007; Mikolov et al., 2013; Mikolov et al., 2010; Pennington et al., 2014).Source: §2 · p. 2
  • Multi-sense models augment embeddings with sense-specific vectors, disambiguating tokens from local co-occurrence contexts.For example, “apple” can be associated with fruit-related contexts or with IT-related contexts.Source: §2 · p. 2
  • Prior methods derive senses through context clustering, non-parametric splitting, topic models, external sense inventories, or part-of-speech-specific representations.Reisinger and Mooney (2010) and Huang et al. (2012) pre-cluster contexts; Neelakantan et al. (2014) relaxes fixed sense counts, while Liu et al. (2015) combines neural embeddings with LDA. Wu and Giles (2015) cluster Wikipedia documents; Chen et al. (2014) use WordNet senses, and Qiu et al. (2014) separate part-of-speech representations.Source: §2 · p. 2
  • Multi-sense methods have mostly been evaluated on the relatively artificial task of matching human word-similarity judgments.Source: §2 · p. 2

3 Learning Sense-Specific Embeddings

The section introduces a CRP-based approach that learns multiple sense-specific embeddings per word and incorporates sense induction into distributed language models; it reports improved performance on a standard word-similarity task.

3 Learning Sense-Specific Embeddings

  • The proposed CRP-based model improves over the state of the art on a standard word-similarity task.Source: §3 · p. 3

3.1 Chinese Restaurant Processes

  • CRP assigns occurrences to existing or new senses based on contextual semantic relatedness and sense popularity, favoring popular senses while avoiding a fixed threshold.Source: §3 · pp. 2–3, §3.1 · p. 3

3.2 Incorporating CRP into Distributed Language Models

  • Each word has a K-dimensional global embedding and a set of discovered senses, each with its own sense-specific embedding.Source: §3.2 · p. 3
  • The model predicts tokens using global vectors and context, then uses CRP to assign or create a sense, jointly learning sense clusters and embeddings.When right-context labels are unavailable, the model uses the right-context words’ global vectors.Source: §3.2 · pp. 3–4

4 Obtaining Word Representations for NLU tasks

The section treats sense labeling in context as inference with global and sense-specific embeddings fixed. Because global optimization over all token senses can be expensive, it uses two simplified heuristic approaches.

  • With embeddings fixed, sense labeling for tokens in context is treated as an inference procedure.Source: §4 · p. 4
  • Finding the globally optimal labeling requires searching all possible senses for all words and can be expensive.Source: §4 · p. 4
  • Greedy search assigns each token its locally optimal sense label and represents it with the associated embedding.Source: §4 · p. 4
  • The expectation approach computes each possible sense's probability and represents the word with the expectation vector.Source: §4 · p. 4

5 Word Similarity Evaluation

On contextual word-similarity judgments in SCWS, multi-sense embeddings outperform the one-vector baseline, with SG+Expect showing the largest reported gains.

5 Word Similarity Evaluation

  • The evaluation uses SCWS, where human similarity judgments are tied to word pairs in context, unlike context-free datasets such as WS353 and RG.Source: §5 · p. 4
  • Models trained on 1.1-billion-token Wikipedia and 120-billion-token combined corpora were iterated three times with window size 11, then compared using cosine similarity.Greedy or Expectation strategies produced context-conditioned token vectors for the similarity calculation.Source: §5 · p. 4
  • On context-dependent SCWS, SG+Greedy and SG+Expect outperform SG; SG+Expect improves over one-word-one-vector by +2.4 on the 1.1 billion Wikipedia dataset and +3.2 on Common Crawl.Source: §5 · pp. 4–5
  • SG+Expect outperforms SG+Greedy because global information aids word-sense selection, whereas neighboring words alone provide insufficient disambiguation.Source: §5 · p. 5
  • Greedy-model nearest-neighbor examples show contextual disambiguation for bank, rock, and apple, while also accommodating less distinct polysemous senses.Source: §5 · p. 5

6 Experiments on NLP Tasks

The experiments compare multi-sense embeddings with single-vector baselines across NLP tasks while controlling embedding dimensionality. Multi-sense embeddings help on some tasks but not others, and their gains depend on task and model.

6 Experiments on NLP Tasks

  • The suite evaluates POS and NER tagging, phrase- and sentence-level sentiment, semantic relationship identification, and sentence-level semantic relatedness, using embeddings trained with word2vec on the same corpus.Experiments use 50d standard skip-gram and sense-disambiguated embeddings, plus 100d concatenations of global and sense-specific vectors compared with 100d doubled-dimensionality skip-gram baselines.Source: §6 · p. 5

6.1 The Tasks

  • The study uses benchmark protocols and standard neural architectures to isolate embedding contributions rather than target state-of-the-art task results.For tagging, neighboring embeddings are concatenated as input features; sentiment models use recurrent or recursive sentence representations.Source: §6 · p. 5, §6.1 · p. 5, Sentence-level Sentiment… · p. 6

Sentence-level Sentiment Classification (Pang)

  • Sentiment experiments fix the input word embeddings and use recurrent models for sentence-level classification and recursive models for treebank constituents.The Pang dataset labels sentences, while the Stanford Sentiment Treebank supplies constituent-level labels; the reported comparisons show no significant improvement from multi-sense embeddings.Source: Sentence-level Sentiment… · p. 6, Table 5 · p. 6, Table 6 · p. 7

6.2 Discussions

  • At matched dimensionality, Expectation often outperforms single-vector baselines, but only some differences are significant; concatenating global and sense-specific vectors is not a fair comparison with 50d embeddings.The concatenation doubles dimensionality to 100d, so the appropriate single-vector baseline is 100d.Source: §6.2 · p. 8
  • Multi-sense embeddings significantly help POS tagging, semantic relationship identification, and semantic relatedness, but do not significantly improve NER or sentiment analysis.The discussion attributes POS gains to the relationship between word senses and POS tags; NER is dominated by the “O” label, while sentiment often depends on words whose senses share sentiment values.Source: Table 3 · p. 6, Table 4 · p. 6, Table 7 · p. 7
  • For semantic relatedness, multi-sense gains disappear with the more sophisticated LSTM model.The discussion suggests neural composition can extract relevant information from mixed single-vector representations, narrowing the gap between embedding types.Source: Table 9 · p. 7, §6.2 · p. 8

7 Conclusion

The paper proposes Chinese-restaurant-process multi-sense embeddings that achieve state-of-the-art performance on simple word-similarity tasks and introduces a pipeline for evaluating them in NLP applications. Results suggest that increasing baseline embedding dimensionality can sometimes match multi-sense gains, while multi-sense embeddings consistently improve some tasks in the tested pipeline.

  • The paper proposes Chinese-restaurant-process multi-sense embeddings with state-of-the-art performance on simple word-similarity tasks and a pipeline for NLP applications.It examines multiple NLP tasks to assess when multi-sense embeddings introduce performance boosts.Source: §7 · p. 8
  • 5GREEDY models may make mistakes because they work aggressively, lack global optimality, and use limited context information.Source: §7 · p. 8
  • Increasing baseline skip-gram embedding dimensionality is sometimes sufficient to match the performance gains from multi-sense embeddings.The conclusion calls increasing dimensionality the most straightforward way to improve performance on these tasks.Source: §7 · p. 9
  • The proposed multi-sense embeddings improve part-of-speech tagging and semantic relation identification, though conclusions are based on the introduced pipeline.Other sense-learning or labeling models, or a different pipeline, may show stronger effects.Source: §7 · p. 9
  • The results underscore the importance of testing embedding models in real NLP applications, beyond simple human-matching tasks.Source: §7 · p. 9

8 Appendix

The appendix describes recurrent and recursive networks for mapping variable-length token sequences to sentence- or phrase-level vectors, and specifies an LSTM variant.

Recurrent Networks

  • A recurrent network combines each word vector with the previous hidden vector, and its final hidden vector represents the whole sequence.The network updates the hidden vector at each timestep and passes it to the next step.Source: §8 · p. 9

Recursive Networks

  • A recursive network builds parent representations bottom-up from neighboring words ordered by a parse tree, combining each node’s immediate children.The recursion continues until it reaches the tree root.Source: §8 · p. 9

Long Short Term Memory (LSTM)

  • LSTMs use input, memory, and output gates at each timestep to compute hidden vectors from the current input and previous hidden vector.The passage distinguishes input-unit vectors e_t from LSTM-computed vectors h_t.Source: §8 · p. 9

http://arxiv.org/ps/1506.01070v3

The supplied passages indicate that figures 2–5 are available in PNG format.

Related papers

Loading 1506.01070v3…