Source-linked AI summary
Self-supervised Learning for Large-scale Item Recommendations
Tiansheng Yao, Xinyang Yi, Derek Zhiyuan Cheng, Felix Yu, Ting Chen, Aditya Menon, Lichan Hong, Ed H. Chi, Steve Tjoa, Jieqi Kang, Evan Ettinger
TL;DR
Large-scale recommenders face severe long-tail label sparsity because feedback is concentrated on a small fraction of millions-to-billions of items. The paper introduces a multi-task SSL framework with feature-correlation-based augmentation for sparse categorical inputs, and reports stronger offline regularization results plus improved live business metrics, especially on under-supervised slices.
Problem
Power-law feedback distributions leave long-tail items with extremely sparse supervision in large catalogs.
Method
A multi-task SSL framework jointly trains recommendation objectives with contrastive tasks built from masked categorical-feature views and feature-correlation-based augmentation.
Results
SSL significantly improves performance when labels are scarce, consistently outperforms non-SSL regularization offline, and improves top-tier metrics in live A/B testing, especially on under-supervised slices.
Takeaways & Limitations
The proposed SSL framework improves item representation learning and generalization for large-scale neural recommendation systems within the evaluated offline and live settings.
Takeaways & Limitations
Future work will investigate how different training schemes affect model quality.
Abstract
from arXiv · showhide
Large scale recommender models find most relevant items from huge catalogs, and they play a critical role in modern search and recommendation systems. To model the input space with large-vocab categorical features, a typical recommender model learns a joint embedding space through neural networks for both queries and items from user feedback data. However, with millions to billions of items in the corpus, users tend to provide feedback for a very small set of them, causing a power-law distribution. This makes the feedback data for long-tail items extremely sparse. Inspired by the recent success in self-supervised representation learning research in both computer vision and natural language understanding, we propose a multi-task self-supervised learning (SSL) framework for large-scale item recommendations. The framework is designed to tackle the label sparsity problem by learning better latent relationship of item features. Specifically, SSL improves item representation learning as well as serving as additional regularization to improve generalization. Furthermore, we propose a novel data augmentation method that utilizes feature correlations within the proposed framework. We evaluate our framework using two real-world datasets with 500M and 1B training examples respectively. Our results demonstrate the effectiveness of SSL regularization and show its superior performance over the state-of-the-art regularization techniques. We also have already launched the proposed techniques to a web-scale commercial app-to-app recommendation system, with significant improvements top-tier business metrics demonstrated in A/B experiments on live traffic. Our online results also verify our hypothesis that our framework indeed improves model performance even more on slices that lack supervision.
1 INTRODUCTION
Large-scale recommendation models must retrieve relevant items from huge catalogs despite highly skewed feedback and sparse supervision for long-tail items. This paper addresses that challenge with a self-supervised framework for sparse categorical features, feature-correlation-based augmentation, and multi-task training, reporting stronger offline and live results.
- Motivation: Power-law interactions concentrate feedback on popular items, leaving training data for long-tail catalog items extremely sparse.Recommendation models also receive relatively little explicit feedback, such as ratings or relevance scores.
- Proposed framework: The framework masks categorical input information, encodes augmented examples with a two-tower DNN, and applies contrastive learning so views of the same example are distinguished from others.The approach adapts self-supervision to sparse, high-cardinality recommendation features rather than dense CV or NLU inputs.
- Proposed framework: The paper proposes a model architecture-agnostic SSL framework that jointly optimizes auxiliary self-supervised and primary supervised losses for sparse neural recommenders.The framework targets efficient item retrieval in large catalogs.
- Proposed framework: The proposed data augmentation exploits correlations among heterogeneous categorical features when constructing masked views.This design is presented as a novel augmentation method tailored to recommender models.
- Evaluation: Offline experiments on one public and one industry-scale recommendation dataset show SSL significantly improves performance, especially when labels are scarce, and consistently beats non-SSL regularization techniques.The paper also reports gains where non-SSL regularization provides no additional improvement.
- Evaluation: Live A/B testing in a web-scale app-to-app recommender shows significantly improved top-tier metrics, with larger gains on slices having little supervision.The deployed system uses a two-tower recommendation model.
2 RELATED WORK
Related work spans self-supervised and unsupervised representation learning in vision, language, and sequential recommendation, along with spread-out regularization and efficient two-tower retrieval models. The paper differs by tailoring self-supervision to sparse categorical recommendation features without requiring a separate data source.
- Self-supervised Learning and Pre-training: Vision SSL methods such as SimCLR use image-specific augmentations, whereas this paper designs augmentations for categorical recommendation features.Examples of image-specific transformations include cropping, rotation, and color distortion.
- Self-supervised Learning and Pre-training: Prior NLU work finds that pre-training tasks aligned with the final task can outperform generic objectives, while this paper uses no separate data source.The comparison concerns dual-encoder models and large-scale public NLU content.
- Spread-out Regularization: Spread-out regularization promotes separation among instances or embeddings to improve generalization, but it does not constitute the paper’s explicit SSL task construction.The paper positions these methods as the closest recommendation-related regularization line.
- Neural Recommenders: Two-tower recommenders efficiently retrieve items by computing query-item dot products and reducing top-k search to maximum-inner-product search.Related systems construct item embeddings from IDs and categorical metadata, unlike models relying only on item IDs that suffer from cold-start issues.
- Self-supervised Learning in Sequential Recommendations: Sequential recommendation research applies SSL to user histories and disentangled user representations, addressing a different focus from this paper’s item-representation framework.The cited work also combines SSL with reinforcement learning to model long-term user interest.
3 METHOD
The method adds a contrastive self-supervised task to two-tower recommendation models, using feature-correlated augmentations to learn robust item representations alongside the supervised objective.
- 3.1 Framework: Contrastive learning encodes two augmented views of each example and pulls representations of the same example together while separating different examples.The resulting embedding space places augmented views of similar items closer and pushes random examples farther apart.
- 3.1 Framework: When augmentations and encoders are identical, the SSL objective reduces to a spread-out-style regularizer, while specific augmentations can outperform spread-out regularization.The paper connects the contrastive formulation to prior spread-out regularization but retains richer augmentation choices.
- 3.2 A Two-stage Data Augmentation: Correlated Feature Masking creates augmented item views by masking highly correlated categorical features together according to mutual information.The method is designed to avoid overly easy contrastive tasks caused by correlated features remaining visible across views.
- 3.2 A Two-stage Data Augmentation: The two-stage augmentation combines masking with dropout, using default embeddings for masked features and randomly removing values from multi-valued categorical features.Dropout further reduces input information and increases the difficulty of the SSL task.
- 3.3 Multi-task Training: The framework jointly optimizes supervised recommendation and auxiliary self-supervised losses to improve item representations for sparse, large-vocabulary recommendation models.The item tower and sparse feature embeddings support both tasks, allowing SSL representations to aid the primary objective.
- 3.3 Multi-task Training: Sampling SSL items uniformly from the corpus avoids bias toward head items created by the power-law training distribution.The main supervised task uses query-item training data, whereas SSL uses a uniform item distribution; the authors report this heterogeneous sampling is critical for superior performance.
4 OFFLINE EXPERIMENTS
Offline experiments evaluate the SSL framework on two large-scale item-to-item recommendation datasets, testing overall quality, sparsity, parameter effects, and augmentation choices. Across these evaluations, CFM outperforms non-SSL regularization, gains more under sparse supervision, and benefits from correlated feature masking.
- 4.3 Effectiveness of SSL with Correlated Feature Masking: CFM consistently performs best against non-SSL regularization on the full Wikipedia and AAI datasets.The comparison answers whether the proposed SSL framework improves recommendation models and shows that data augmentation is important for SSL regularization.
- 4.3 Effectiveness of SSL with Correlated Feature Masking: SSL-trained app embeddings form clearer category-based clusters than baseline embeddings in t-SNE visualizations.The stronger semantic structure partially explains the observed performance gains.
- 4.4 Data Sparsity: Under 10% training data, CFM improves Recall@10 by 20.6% on Wikipedia and 25.7% on AAI relative to baseline.On the full datasets, the corresponding improvements are 6.1% and 10.2%; CFM also increasingly outperforms feature dropout as data becomes sparser.
- 4.5 Comparison of Different Data Augmentations: Among augmentation variants, CFM outperforms all alternatives, while mutual-information-based masking, complementary views, and feature dropout each contribute to performance.Random masking causes the largest drop, and removing either complementary masking or the second dropout phase reduces performance.
5 LIVE EXPERIMENT
The proposed SSL framework was evaluated in a web-scale commercial app recommender through live A/B testing, improving overall business and engagement metrics. Gains were larger for fresh apps and international markets with sparser training data.
- +1.5% top business metric and +0.67% key user engagement were achieved over 14 days in the overall live experiment.The deployed model added the best-performing SSL task to the existing production two-tower DNN.
- +5.47% top business metric improvement occurred in international countries with sparser training data.This slice result supports the hypothesis that SSL helps where supervision is limited.
- +4.5% user-engagement improvement occurred for fresh apps, a cold-start slice with limited supervision.The result was measured during the same 14-day live A/B test.
- The SSL-empowered model was successfully launched in the production recommendation system after the live experiment.
6 CONCLUSION
The paper concludes that its architecture-agnostic SSL framework improves large-scale neural recommender models and introduces a feature-correlation-aware augmentation for heterogeneous categorical features. It identifies alternative training schemes and extensions to other application domains as future work.
- The paper proposes a model architecture agnostic SSL framework for large-scale neural recommender models.
- Its novel data augmentation targets heterogeneous categorical features and outperforms other evaluated variants.
- Future work will investigate how different training schemes affect model quality, including SSL pre-training followed by supervised fine-tuning.
- The technique could also be extended to deep models for search ranking or pCTR prediction.
A APPENDIX
The Wikipedia and AAI datasets exhibit highly skewed item-frequency distributions, with a small number of items accounting for substantial portions of training data.
- The top 50 items in AAI collectively appear in roughly 10% of training examples.The appendix uses the item-frequency CDF to characterize dataset skew.
Dataset Statistics.
The appendix evaluates a naive popularity recommender and partitions test examples into head and tail groups according to ground-truth item frequency and training-set presence.
- Dataset Statistics.: TopPopular achieves Recall@50 ≈0.1 on AAI and Recall@50 ≈0.05 on Wikipedia.The CDF of the K-th frequent item corresponds to Recall@K for this baseline.
- Dataset Statistics.: All proposed methods outperform the TopPopular baseline by a large margin.
- Dataset Statistics.: AAI defines Head as ground-truth items in the top 10% by frequency, while Wikipedia treats unseen test items as Tail and the rest as Head.
Head and Tail Item Evaluation Dataset.
The evaluation visualizes app embeddings from AAI-trained models with t-SNE to assess representation quality. The best SSL model shows apps tending to group better by category than the non-SSL comparison.
- The best SSL model’s app embeddings tend to group better by category than those in the non-SSL comparison.The comparison uses app embeddings learned from models trained on the AAI dataset and is shown in Figure 4.
- Figure 4 uses t-SNE plots of app embeddings to evaluate whether SSL learns higher-quality representations.
- Apps from different categories are plotted in different colors, with the category mapping provided in the figure legends.