Source-linked AI summary
Word2Vec applied to Recommendation: Hyperparameters Matter
Hugo Caselles-Dupré, Florian Lesaint, Jimena Royo-Letelier
TL;DR
Recommendation applications often reuse SGNS hyperparameters tuned for NLP despite differences in data and tasks. This paper searches hyperparameter grids across four recommendation datasets and finds that optimizing neglected parameters substantially improves performance, with recommendation-optimal settings differing from NLP settings.
Problem
Recommendation studies using SGNS rarely discuss hyperparameters already tuned for NLP, despite hyperparameters being data- and task-dependent and recommendation data and tasks differing from NLP.
Method
The study performs large SGNS hyperparameter grid searches across four recommendation datasets and evaluates item embeddings using Next Event Prediction.
Results
Optimizing negative sampling distribution, epochs, subsampling, and window size significantly improves recommendation performance; on Click-Stream, optimization increases performance by a factor of 10.
Takeaways & Limitations
Recommendation-optimal SGNS configurations differ noticeably from NLP configurations, particularly for negative sampling, whose optimal distribution can favor unpopular negative items.
Takeaways & Limitations
The study assumes that works not discussing the negative-sampling parameter rely on its commonly accepted default value.
Abstract
from arXiv · showhide
Skip-gram with negative sampling, a popular variant of Word2vec originally designed and tuned to create word embeddings for Natural Language Processing, has been used to create item embeddings with successful applications in recommendation. While these fields do not share the same type of data, neither evaluate on the same tasks, recommendation applications tend to use the same already tuned hyperparameters values, even if optimal hyperparameters values are often known to be data and task dependent. We thus investigate the marginal importance of each hyperparameter in a recommendation setting through large hyperparameter grid searches on various datasets. Results reveal that optimizing neglected hyperparameters, namely negative sampling distribution, number of epochs, subsampling parameter and window-size, significantly improves performance on a recommendation task, and can increase it by an order of magnitude. Importantly, we find that optimal hyperparameters configurations for Natural Language Processing tasks and Recommendation tasks are noticeably different.
1 INTRODUCTION
Word2Vec methods transfer successfully from NLP to recommendation, but recommendation data and objectives differ from language, making inherited hyperparameter defaults questionable. The paper therefore studies hyperparameter importance for SGNS recommendation settings.
- Word2Vec creates low-dimensional word representations and has been applied to recommendation to generate scalable user and product embeddings.The method can scale to millions of items.
- Language corpora and item sequences have different structures, while NLP and recommendation tasks emphasize different item or word frequency patterns.Popular items matter for recommendation, whereas frequent linking words are less relevant to linguistic tasks.
- Recommendation studies rarely discuss SGNS hyperparameter values despite defaults being tuned for NLP tasks such as word similarity and analogy detection.Widely used implementations commonly set these values by default.
- The study searches SGNS hyperparameters across four recommendation dataset types using Next Event Prediction as an offline recommendation proxy.It identifies negative sampling distribution, epochs, subsampling, and window-size as important variables.
- The paper reports that recommendation-optimal configurations are radically different from NLP configurations, especially for negative sampling distribution.The work evaluates marginal hyperparameter importance through large grid searches.
2 WORD2VEC
SGNS learns item or word vectors from contextual co-occurrences while contrasting observed contexts with sampled negatives. Its recommendation performance depends on several hyperparameters whose conventional values largely originate from NLP practice.
- SGNS method: SGNS represents each word as a vector and brings vectors closer when words appear in similar contexts.The method is based on the Distributional Hypothesis.
- SGNS method: For each observed pair, SGNS samples k negative examples from an α-smoothed unigram distribution.The loss maximizes observed-context scores and minimizes scores for negative examples.
- Negative sampling: The negative-sampling parameter α controls which items are sampled: α = 1 follows frequency, α = 0 is uniform, and negative α favors unpopular items.The conventional NLP setting is α = 0.75.
- Hyperparameter motivation: The study identifies epochs, window-size, subsampling, and negative sampling distribution as hyperparameters whose simultaneous optimization improves recommendation performance.The investigated literature rarely discusses subsampling and commonly inherits fixed defaults for several variables.
- Context and subsampling: The window-size determines the maximum sequence gap for shared context, while subsampling randomly removes sufficiently frequent items.Dynamic windows and subsampling can improve performance and accelerate training.
3 EXPERIMENTS
The experiments evaluate SGNS item embeddings on four recommendation datasets using Next Event Prediction and large hyperparameter searches. They compare dataset types, configurations, and recommendation-specific alternatives under a common evaluation procedure.
- Experimental design: The study evaluates seven SGNS hyperparameters on four recommendation datasets from diverse sources.The datasets include music, e-commerce, and click-stream data.
- Datasets: The music data comprise 100,000 sampled sessions each, with sharp popularity-distribution tails and similar count distributions.The two datasets are 30Music and Deezer.
- Datasets: The Online Retail dataset contains 4,234 purchase histories and has a heavier distribution tail than the music datasets.Its popular-versus-unpopular item discrepancy is smaller.
- Datasets: The kosarak dataset contains 83,625 anonymized click-stream histories, with a distribution tail comparable to the music datasets.It comes from a Hungarian online news portal.
- Task and metrics: Next Event Prediction trains on earlier sequence items, tunes hyperparameters on validation pairs, and predicts the next item using nearest neighbors.The test pairs are disjoint from validation pairs.
- Hyperparameter search: The search evaluates epochs, window-size, subsampling, negative sampling, embedding size, negative samples, and learning rate across 300k models.The three latter variables contributed less than 2% and were fixed to default values for the main analysis.
4 RESULTS
Hyperparameter optimization substantially improves SGNS recommendation performance, while the best negative-sampling choice differs from the NLP default. Gains are especially large for click-stream data and remain relevant for recommendation-specific Meta-Prod2vec.
- Hyperparameter search roughly doubles performance on both Music datasets, increases Click-Stream performance tenfold, and yields substantial gains on E-commerce.
- The optimal negative-sampling parameter is α = −0.5 for 30Music, unlike the NLP value α = 0.75, and negative α is also optimal for Deezer and Click-Stream.Negative α samples unpopular items more often as negative examples.
- Including α in the hyperparameter search significantly improves final performance across all datasets.
- Meta-Prod2vec also benefits from hyperparameter optimization with a negative α, but optimized SGNS outperforms it.
- The results indicate that optimal SGNS hyperparameters depend on both the dataset and the recommendation task.
5 CONCLUSION
The paper concludes that recommendation data and tasks require reconsidering Word2vec defaults inherited from NLP. Across four datasets, optimizing four hyperparameters improves performance, but the best settings vary with data distributions and item-popularity structure.
- Optimizing negative sampling distribution, epochs, subsampling, and window-size significantly improves recommendation performance across four datasets.
- Different recommendation data distributions produce different optimal hyperparameter values on the same Next Event Prediction task.
- Popularity homogeneity within sequences, popularity-distribution shape, and catalog heterogeneity affect task evaluation and optimal settings.
- The paper leaves open whether optimal hyperparameters can be inferred from data structure.The authors identify this as a direction that could improve SGNS-based recommendation results.