PaperlayerView exact revision on arXiv

Source-linked AI summary

PTE: Predictive Text Embedding through Large-scale Heterogeneous Text Networks

Jian Tang, Meng Qu, Qiaozhu Mei

arXiv:1508.00200v1cs.CLcs.LGcs.NE

TL;DR

Unsupervised text embeddings are scalable and broadly applicable but are not specifically tuned to labeled tasks, where CNN-based approaches can perform better. PTE addresses this gap by embedding labeled and unlabeled co-occurrence information in a heterogeneous text network; it outperforms unsupervised embeddings and is competitive with CNNs, with efficiency advantages. Its short-document performance is not consistently superior to CNNs, partly because it does not use word order.

  • Problem

    Unsupervised embeddings are general but have weaker predictive power for particular tasks because they do not use available labeled information.

  • Method

    PTE jointly embeds word-word, word-document, and word-label networks constructed from labeled and unlabeled data into a low-dimensional text representation.

  • Results

    PTE significantly outperforms state-of-the-art unsupervised embeddings across text-classification tasks and outperforms CNNs on long documents while matching them on short documents.

  • Takeaways & Limitations

    PTE offers task-oriented text embeddings with greater efficiency, effective use of unlabeled data, and fewer parameters to tune than CNNs.

  • Takeaways & Limitations

    On short documents, PTE does not consistently outperform CNNs because it does not leverage word order, leaving room for improvement.

Abstract

from arXiv · show

Unsupervised text embedding methods, such as Skip-gram and Paragraph Vector, have been attracting increasing attention due to their simplicity, scalability, and effectiveness. However, comparing to sophisticated deep learning architectures such as convolutional neural networks, these methods usually yield inferior results when applied to particular machine learning tasks. One possible reason is that these text embedding methods learn the representation of text in a fully unsupervised way, without leveraging the labeled information available for the task. Although the low dimensional representations learned are applicable to many different tasks, they are not particularly tuned for any task. In this paper, we fill this gap by proposing a semi-supervised representation learning method for text data, which we call the \textit{predictive text embedding} (PTE). Predictive text embedding utilizes both labeled and unlabeled data to learn the embedding of text. The labeled information and different levels of word co-occurrence information are first represented as a large-scale heterogeneous text network, which is then embedded into a low dimensional space through a principled and efficient algorithm. This low dimensional embedding not only preserves the semantic closeness of words and documents, but also has a strong predictive power for the particular task. Compared to recent supervised approaches based on convolutional neural networks, predictive text embedding is comparable or more effective, much more efficient, and has fewer parameters to tune.

1. INTRODUCTION

PTE addresses the task-specific weakness of unsupervised text embeddings by combining labeled and unlabeled information in an efficient heterogeneous-network embedding. Experiments report stronger classification performance than unsupervised embeddings and competitive results with CNNs, while retaining efficiency and ease of tuning.

  • Text representations are critical for classification, clustering, and retrieval, but independent word and bag-of-words representations ignore semantic relatedness and suffer sparsity, polysemy, and synonymy.
  • Unsupervised embeddings generalize across tasks but usually have weaker task-specific predictive power than labeled deep-learning representations.
  • PTE jointly learns low-dimensional text representations from limited labeled examples and abundant unlabeled examples, adapting unsupervised embeddings to particular tasks.
  • The method embeds word-word, word-document, and word-label co-occurrence networks, and represents arbitrary text by averaging its word embeddings.The optimization scales to millions of documents and billions of tokens on a single machine.
  • PTE significantly outperforms state-of-the-art unsupervised embeddings across text-classification tasks and outperforms CNNs on long documents while producing comparable short-document results.The paper also reports greater efficiency, effective use of unlabeled data, and lower sensitivity to parameter settings than CNNs.

2. RELATED WORK

Related work spans unsupervised and supervised distributed text representations, sentiment-specific embeddings, and large-scale graph embedding. The central distinction is whether labels enter representation learning directly, alongside differences in scalability and task generality.

  • Distributed text representations support tasks including word analogy, POS tagging, parsing, language modeling, and sentiment analysis.
  • Unsupervised methods learn from local or document-level word co-occurrences and scale to millions of documents, whereas supervised methods commonly use deep neural architectures.
  • Unsupervised methods exclude labels during representation learning, while RNTNs and CNNs incorporate labels directly to tune representations for the task.
  • A sentiment-specific word-vector approach is limited by its inability to scale to millions of documents or generalize to other classification tasks.
  • PTE is related to graph embedding because it learns word representations through a heterogeneous text network, unlike classical methods unsuitable for very large networks.

3. PROBLEM DEFINITION

PTE defines task-oriented text embedding by combining multiple co-occurrence networks built from labeled and unlabeled text. The resulting heterogeneous network is embedded into a low-dimensional space to represent words and, by extension, larger text units.

  • The goal is to learn low-dimensional text representations optimized for a given classification task rather than only general semantic representations.
  • The word-word network captures local-context co-occurrences in unlabeled data, with edge weights counting co-occurrences within a specified window.
  • The word-document network captures document-level co-occurrences as a weighted bipartite graph linking words to documents.
  • The word-label network encodes labeled information through category-level word co-occurrences, weighted by word frequencies in documents assigned to each class.
  • These word-word, word-document, and word-label networks are integrated into one heterogeneous text network for embedding.
  • The framework can be generalized to word-sentence, word-paragraph, and document-label networks, although this work focuses on word networks.

4. PREDICTIVE TEXT EMBEDDING

PTE learns predictive text embeddings by embedding heterogeneous text networks that combine multiple word co-occurrence relations and labeled information. It preserves second-order proximity, supports joint or pre-training-plus-fine-tuning strategies, and averages learned word vectors to represent text.

  • 4.1 Bipartite Network Embedding: PTE first embeds heterogeneous text networks into low-dimensional word representations, then infers embeddings for arbitrary text.The network consists of word-word, word-document, and word-label relations.
  • 4.1 Bipartite Network Embedding: The bipartite embedding model preserves second-order proximity by matching conditional vertex distributions to empirical distributions using KL-divergence.Vertex importance can be weighted by degree, and the objective is optimized with stochastic gradient descent, edge sampling, and negative sampling.
  • 4.2 Heterogeneous Text Network Embedding: The heterogeneous network shares word vertices across word-word, word-document, and word-label networks, whose embeddings are learned with the same bipartite model.This provides a common word representation across the three relation types.
  • 4.2 Heterogeneous Text Network Embedding: PTE supports joint training on labeled and unlabeled networks or pre-training on unlabeled networks followed by fine-tuning with the word-label network.The joint-training implementation samples edges from all three network types and updates the corresponding embeddings.
  • 4.3 Text Embedding: A text embedding is obtained by averaging the learned embeddings of the words it contains.The average is also the closed-form solution to minimizing Euclidean distance between word and text embeddings, unlike the paragraph-vector loss described here.

5. EXPERIMENTS

The experiments evaluate PTE across text classification tasks using long- and short-document corpora. The datasets include 20newsgroups, Wikipedia, DBLP titles, movie reviews, and Twitter data, with minimal preprocessing.

  • Long and Short Document Corpora: The evaluation covers a variety of text classification tasks and datasets.The experiments include both long- and short-document settings.
  • Long Document Corpora: Long-document corpora include 20newsgroups and a Wikipedia snapshot containing around two million English articles.The Wikipedia classification task uses seven diverse categories, while 20newsgroups contains 20 categories.
  • Short Document Corpora: Short-document corpora include DBLP paper titles, one-sentence movie reviews, and Twitter data for sentiment classification.The Twitter corpus contains a randomly sampled 1,200,000 Tweets split into training and testing sets.
  • Data Processing: The original data receive no further normalization such as stop-word removal or stemming.Detailed dataset statistics are summarized in Table 1.

Compared Algorithms

PTE is compared with bag-of-words, unsupervised embedding methods, paragraph-vector variants, LINE, and supervised CNN-based text embedding. Its variants differ in which network types they use and whether labeled information is integrated jointly or through fine-tuning.

  • Compared Algorithms: The comparison includes bag-of-words, Skip-gram, PVDBOW, PVDM, LINE, CNN, and PTE.These baselines span classical representations, unsupervised embeddings, network embedding, and supervised neural approaches.
  • Compared Algorithms: BOW represents each document with a |V|-dimensional TFIDF-weighted vector.This is the classical bag-of-words representation used in the experiments.
  • Compared Algorithms: Skip-gram produces document embeddings by averaging its word embeddings, while PVDBOW ignores word order and PVDM considers it.The paragraph-vector variants differ in whether document word order is modeled.
  • Compared Algorithms: LINE learns unsupervised embeddings from word-word, word-document, or combined networks.It is the large-scale information network embedding baseline.
  • Compared Algorithms: CNN is adapted from sentence modeling to general word sequences and can use unsupervised pre-training, producing the CNN(pretrain) variant.Standard CNN typically uses fully labeled documents.
  • Compared Algorithms: PTE variants use word-label information alone, pre-training plus fine-tuning, or joint training across word-word, word-document, and word-label networks.They are denoted PTE(Gwl), PTE(pretrain), and PTE(joint), respectively.

Classification and Parameter Settings

The classification experiments compare predictive text embeddings with unsupervised embeddings and CNNs across long and short documents. Results show consistent benefits from joint labeled–unlabeled training, while PTE is less consistently superior on short documents.

  • Parameter Settings: Training uses held-out test data, while labeled training documents contribute labels during PTE representation learning but not during unsupervised embedding learning.Classification uses one-vs-rest logistic regression and reports micro-F1 and macro-F1.
  • Long Documents: PTE jointly trained with heterogeneous networks or word-document and word-label networks performs best among approaches on long documents.The joint model outperforms PTE using only the word-label network and PTE pretraining.
  • Long Documents: Jointly training with labeled and unlabeled data is more effective than separating pretraining and fine-tuning, and PTE(joint) consistently outperforms CNN on long documents.Even CNNs initialized with unsupervised embeddings remain inferior to PTE(joint).
  • Long Documents: PTE(joint) is more effective than PTE(pretrain) on RCV1, reinforcing the advantage of jointly using labeled and unlabeled data.Predictive embeddings also outperform unsupervised embeddings on the RCV1 long-document datasets.
  • Efficiency: PTE(joint) is typically more than 10 times faster than CNN models and remains more than 5 times faster when CNN is pretrained.The comparison is made on the imdb dataset.
  • Short Documents: On short documents, PTE improves over corresponding unsupervised embeddings, but CNN achieves the best result on twitter and PTE(joint) does not consistently outperform CNN.Document-level co-occurrences suffer from sparsity in short documents, and PTE does not use word order.

5.3 Effects of Labeled Data

As labeled data increases, both CNN and PTE performance generally improves, but their relative advantage depends on document type and label availability. Extremely scarce labels can make PTE vulnerable to noisy word-label information.

  • Both CNNs and PTEs improve as the size of labeled data increases.
  • With very few training examples, unsupervised-embedding pretraining can make CNN outperform all PTE variants, especially on short documents.
  • When labels are scarce, PTE can underperform Skip-gram on dblp because the word-label network becomes noisy while receiving equal treatment with word-word and word-document networks.
  • The authors leave adjusting sampling probabilities across network types under scarce-label conditions as future work.

5.4 Effects of Unlabeled Data

The unlabeled-data analysis compares CNN pretraining with PTE, which can use unlabeled data for pretraining or joint training, on 20ng and dblp.

  • CNN uses unlabeled data for pretraining, whereas PTE can use it for either pretraining or joint training.
  • The comparison evaluates performance against unlabeled-data size on the 20ng and dblp data sets.
  • On 20ng, 10% of documents are labeled and the remainder are treated as unlabeled; on dblp, 200,000 titles from other conferences are sampled as unlabeled data.

5.5 Parameter Sensitivity

PTE(joint) is tested for sensitivity to the edge-sampling sample count T on 20ng and dblp. Its performance converges once T becomes sufficiently large.

  • PTE(joint) performance is analyzed as a function of the number of edge samples T on 20ng and dblp.
  • On both data sets, PTE(joint) performance converges when T becomes large enough.
  • In practice, T can be set sufficiently large because most other PTE parameters are not sensitive across data sets.

5.6 Document Visualization

The document visualization compares unsupervised and predictive embeddings on 20ng using t-SNE. Predictive embeddings separate document classes more clearly in both training and test data.

  • The visualization compares unsupervised and predictive document embeddings on the 20ng data set using t-SNE.
  • The unsupervised embedding uses LINE(Gwd), while the predictive embedding uses PTE(Gwl).
  • Predictive embeddings distinguish different classes much better than unsupervised embeddings in both training and test documents.

6. DISCUSSION AND CONCLUSION

The discussion recommends choosing between unsupervised embeddings, CNNs, and PTE according to document length and labeled-data availability. PTE offers efficient joint use of labeled and unlabeled data, while incorporating word order remains an avenue for improvement.

  • Unsupervised embeddings: Document length changes which co-occurrence information is most useful: document-level for long documents and local context for short documents.Combining local and document-level co-occurrences further helps on short documents but not on long documents.
  • Practical guidelines: When no labeled data is available, use LINE(Gwd) for long documents and LINE(Gwd+Gww) for short documents.
  • Practical guidelines: With few labeled examples, pre-train CNN with unsupervised embeddings for short documents and use PTE for long documents.
  • Predictive embeddings: On short documents, choosing between PTE(joint) and CNN or CNN(pretrain) trades performance against efficiency.
  • Discussion and conclusion: PTE remains an efficient direction for predictive distributed text embeddings, with word-order modeling identified as room for improvement.
Loading 1508.00200v1…