Source-linked AI summary
StarSpace: Embed All The Things!
Ledell Wu, Adam Fisch, Sumit Chopra, Keith Adams, Antoine Bordes, Jason Weston
TL;DR
Existing embedding methods are often specialized to particular labeling, ranking, recommendation, graph, or representation tasks. StarSpace embeds and ranks feature-based entities with task-dependent similarities, achieving performance on par with or better than competing methods across several evaluated tasks while remaining broadly applicable.
Problem
Embedding methods are often specialized to particular tasks, while some settings such as ranking unseen documents per user cannot use supervised classification models directly.
Method
StarSpace embeds entities represented as bags of discrete features and ranks them using relationships and task-appropriate positive and negative pair selection.
Results
Across recommendation, knowledge-base link prediction, search, sentence matching, and sentence embedding tasks, StarSpace is on par with or outperforms several competing embedding methods.
Takeaways & Limitations
StarSpace provides a generally applicable baseline for diverse embedding tasks and can train directly on downstream prediction or ranking tasks.
Takeaways & Limitations
StarSpace’s performance depends strongly on choosing suitable positive and negative pair generators, and some broader tasks were not evaluated.
Abstract
from arXiv · showhide
We present StarSpace, a general-purpose neural embedding model that can solve a wide variety of problems: labeling tasks such as text classification, ranking tasks such as information retrieval/web search, collaborative filtering-based or content-based recommendation, embedding of multi-relational graphs, and learning word, sentence or document level embeddings. In each case the model works by embedding those entities comprised of discrete features and comparing them against each other -- learning similarities dependent on the task. Empirical results on a number of tasks show that StarSpace is highly competitive with existing methods, whilst also being generally applicable to new cases where those methods are not.
1 Introduction
StarSpace is a general neural embedding model for labeling, ranking, recommendation, graph embedding, and word, sentence, or document embedding tasks. It embeds discrete-feature entities and compares them in a shared space, achieving competitive results across six evaluated tasks.
- StarSpace targets text classification, entity ranking, collaborative and content-based recommendation, multi-relational graph embedding, and word, sentence, or document embedding.
- StarSpace is on par with or outperforms several competing methods while remaining applicable to cases where many competitors are not.
- The model learns entity embeddings and compares potentially different entity types in a common space for task-specific ranking or classification.
- Six evaluations cover text classification, knowledge-base link prediction, document recommendation, article search, sentence matching, and general sentence embeddings.
2 Related Work
Prior work includes unsupervised and supervised embedding methods for language, classification, retrieval, recommendation, and knowledge-base link prediction. StarSpace extends these embedding ideas across collaborative and content-based recommendation and matches TransE on Freebase link prediction.
- Unsupervised word and sentence embedding methods include word2vec and fastText, while supervised approaches target classification and information retrieval.
- Many collaborative-filtering recommenders use fixed user and item embeddings, limiting natural incorporation of new users or items.
- StarSpace supports both collaborative-filtering and content-based recommendation, allowing natural out-of-sample extensions when users and items are represented as features.
- For knowledge-base link prediction, StarSpace outperforms several methods and matches TransE.
3 Model
StarSpace represents entities as bags of discrete features, embeds them in a shared vector space, and trains task-specific similarity through positive and negative entity pairs. Different generators instantiate classification, recommendation, graph, retrieval, and embedding tasks.
- Entities are represented by bags of discrete features whose vectors are summed to form an embedding, including documents, sentences, users, words, and item IDs.
- Training compares positive entity pairs from a task-dependent generator with k negative samples using a similarity function and batch loss.
- Experiments use margin ranking loss, which performed on par or better than the implemented negative log loss of softmax.
- The model learns a similarity function and uses it at test time for label prediction, ranking, or downstream embedding-based tasks.
- Task-specific generators define examples for classification, collaborative or content-based recommendation, knowledge-graph link prediction, information retrieval, and word or sentence embedding.
- Sentence embeddings can be trained directly from sentence pairs within documents, matching training examples to sentence-length test inputs.
4 Experiments
Across six evaluated tasks, StarSpace serves as a general embedding and ranking system, matching or outperforming competing methods while remaining applicable beyond fixed-label classification.
- Text Classification: In text classification, StarSpace outperforms several methods and performs similarly to fastText.It is also substantially faster than deep learning approaches, although fastText trains faster.
- Document Recommendation: StarSpace directly handles recommendation of variable sets of previously unseen documents, unlike supervised classification models with fixed label sets.It ranks a user’s next article from the user’s preceding clicked-article history.
- Document Recommendation: StarSpace outperforms word2vec, fastText, Tagspace, and SVMs by a significant margin on document recommendation.
- Link Prediction: Embedding Multi-relation Knowledge Graphs: On Freebase 15k link prediction, StarSpace and TransE give similar performance at the same embedding dimension.The evaluation ranks corrupted heads or tails using mean predicted rank and hits@10, including filtered evaluation.
- Wikipedia Article Search & Sentence Matching: On Wikipedia article search and sentence matching, StarSpace outperforms TFIDF and fastText by a significant margin, while sentence-embedding results are strong on many transfer tasks.No method wins outright across all sentence-transfer tasks, but StarSpace is particularly strong on Semantic Textual Similarity.
5 Discussion and Conclusion
StarSpace is presented as a general embedding and ranking system that performs competitively across diverse tasks while remaining broadly applicable. The authors also identify efficiency, representation, and training-data choices as important boundaries and directions for improvement.
- 5 Discussion and Conclusion: StarSpace supports text classification, content-based recommendation, knowledge-base link prediction, search, sentence matching, and sentence embeddings.The paper reports results across these task families, including three SentEval transfer tasks and Freebase 15K link prediction.
- 5 Discussion and Conclusion: StarSpace performs comparably to fastText on text classification and matches TransE on Freebase 15K link prediction.It also outperforms several competing methods on link prediction and achieves results comparable to fastText across three classification datasets.
- 5 Discussion and Conclusion: StarSpace directly handles content-based recommendation and outperforms off-the-shelf fastText, TagSpace, and word2vec on that task.Its feature-based formulation allows direct training for the recommendation objective.
- 5 Discussion and Conclusion: The method’s generality comes from representing labels or ranked entities with features and selecting task-suitable positive and negative examples.The authors note that choosing the wrong positive and negative generators produces greatly inferior results.
- 5 Discussion and Conclusion: Future work targets continuous features, nonlinear representations, image entities, and hierarchical classification to improve efficiency without losing generality.The proposed efficiency enhancement must preserve the model’s generality.