PaperlayerView exact revision on arXiv

Source-linked AI summary

Topic2Vec: Learning Distributed Representations of Topics

Li-Qiang Niu, Xin-Yu Dai

arXiv:1506.08422v1cs.CLcs.LG

TL;DR

LDA’s probability distributions describe occurrence statistics, while embeddings offer another way to represent words and documents. Topic2Vec places topics in the same semantic vector space as words, and its qualitative comparisons with LDA are reported as distinctive and meaningful.

  • Problem

    LDA’s probability distributions describe occurrence statistics, and low-probability words that may distinguish topics are harder to select as representatives.

    Source: §1 · p. 1

  • Method

    Topic2Vec incorporates topics into a Word2Vec-based framework to learn topic and word representations in the same semantic vector space.

    Source: §1 · pp. 1–2, §3 · p. 2

  • Results

    Topic2Vec produces distinctive and meaningful results compared with LDA in evaluations using listed examples and t-SNE visualizations of nearest words.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors conclude that Topic2Vec models topics better than LDA based on their experimental observations.

    Source: §5 · p. 4

    The evaluation is qualitative, and Topic2Vec requires LDA first to assign a topic to each corpus word.

    Source: §5 · pp. 4–5

Abstract

from arXiv · show

Latent Dirichlet Allocation (LDA) mining thematic structure of documents plays an important role in nature language processing and machine learning areas. However, the probability distribution from LDA only describes the statistical relationship of occurrences in the corpus and usually in practice, probability is not the best choice for feature representations. Recently, embedding methods have been proposed to represent words and documents by learning essential concepts and representations, such as Word2Vec and Doc2Vec. The embedded representations have shown more effectiveness than LDA-style representations in many tasks. In this paper, we propose the Topic2Vec approach which can learn topic representations in the same semantic vector space with words, as an alternative to probability. The experimental results show that Topic2Vec achieves interesting and meaningful results.

1 Introduction

LDA represents documents and topics with probability distributions that emphasize occurrence statistics, while low-probability words can be more topic-distinguishing. Topic2Vec incorporates topics into a Word2Vec-based embedding space, and its qualitative comparisons with LDA are reported as distinctive and meaningful.

1 Introduction

  • LDA emphasizes occurrence statistics and may favor frequent words such as “food” over more topic-distinguishing words such as “cheeseburger.”The passage gives analogous examples for drugs and technology.Source: §1 · p. 1
  • Word and document embeddings learn low-dimensional vectors in a shared semantic space, and Word2Vec captures semantic-syntactic relationships between words.The introduction motivates asking whether topics can also be embedded in that space.Source: §1 · p. 1
  • Topic2Vec incorporates topics into a Word2Vec-based neural language-model framework to learn topic representations in the same semantic space as words.The approach is presented as an embedding alternative to probability-based representations.Source: §1 · pp. 1–2
  • Topic2Vec is evaluated against LDA through listed examples and t-SNE visualizations of nearest words, with results described as distinctive and meaningful.Source: §1 · p. 2

2 Related Models

LDA models documents as mixtures of latent topics and estimates document-topic and topic-word probability matrices. Word2Vec learns continuous word vectors using CBOW or Skip-gram objectives based on context.

2.1 Latent Dirichlet Allocation

  • LDA assumes each document is a mixture of latent topics, with each topic represented as a probability distribution over vocabulary words.Source: §2.1 · p. 2
  • Gibbs Sampling estimates document-topic and topic-word probability matrices, and can assign latent topics and word-level topic labels for a new document.Source: §2.1 · p. 2

2.2 Word2Vec

  • Word2Vec learns continuous vector representations of words from large datasets using CBOW or Skip-gram objectives.Source: §2.2 · p. 2
  • In CBOW, context words inform prediction of the current word; the conditional probability p(wj|wi) is calculated with softmax over the vocabulary.The context is denoted wcxt, and k is the context-window size.Source: §2.2 · p. 2

3 Topic2Vec

Topic2Vec extends Word2Vec-style training to learn topic and word representations together, using topic assignments inferred from LDA. Its CBOW and Skip-gram variants predict words and topics in complementary directions.

3 Topic2Vec

  • CBOW predicts the current word and its LDA-assigned topic from surrounding words, while Skip-gram predicts surrounding words from the current word-topic pair.Training uses a document sequence of word-topic pairs, with topics inferred from LDA.Source: §3 · p. 2
  • Topic2Vec learns topic representations alongside word representations in a shared semantic space.It uses negative sampling, stochastic gradient descent, and back-propagation; its complexity is linear in dataset size, as in Word2Vec.Source: §3 · pp. 2–3
  • Figure 2 compares LDA's conditional-probability listings with Topic2Vec's cosine-similarity listings for nearest words and topics.Source: Figure 2 · p. 3

4 Experiments

The experiments compare Topic2Vec and LDA on selected topic words and their two-dimensional embeddings. Topic2Vec’s selected words are reported as more representative, with better topic distinction and word grouping in the embedding.

4.1 Dataset

  • Training uses 100,000 Los Angeles Times documents with more than 1,000 characters each, yielding about 42 million words and a vocabulary of 102,644.Words occurring fewer than five times and stop words were removed.Source: §4.1 · p. 3

4.3 Analysis of Results

  • In the t-SNE visualization, Topic2Vec produces better grouping and separation of words across topics, while LDA words from different topics tend to mix.Source: Figure 3 · p. 4, §4.3 · p. 4

4.2 Evaluation Methods

  • The evaluation compares the top ten conditional-probability words per LDA topic with Topic2Vec words selected by cosine similarity to topic vectors.Topic2Vec represents words and topics as low-dimensional vectors.Source: §4.2 · p. 4

4.3 Analysis of Results

  • For Topics 19 and 27, Topic2Vec distinguishes a drug-focused topic from a treatment-focused topic, whereas LDA returns broader patient- or medical-related words.Source: §4.3 · p. 4
  • Topic2Vec selects more typical and representative words for each topic than LDA and can better distinguish different topics.Source: §4.3 · p. 4

5 Conclusions and Future Work

Topic2Vec combines NPLM, Word2Vec, and LDA to embed latent topics in the same semantic vector space as words; the authors conclude it models topics better than LDA. They note that evaluation is qualitative, Topic2Vec requires prior LDA topic assignment, and they plan to explore independent topic models.

  • Topic2Vec integrates NPLM, Word2Vec, and LDA to embed latent topics in the same semantic vector space as words, and the authors conclude it models topics better than LDA.Source: §5 · p. 4
  • The authors describe their comparison with LDA as qualitative and say Topic2Vec requires running LDA first to assign a topic to each corpus word.Source: §5 · p. 4
  • They plan to explore independent topic models that mine document themes and learn inherent representations to model topics better simultaneously.Source: §5 · pp. 4–5

Related papers

Loading 1506.08422v1…