Controllable Multi-Interest Framework for Recommendation

Yukuo Cen; Jianwei Zhang; Xu Zou; Chang Zhou; Hongxia Yang; Jie Tang

Source-linked AI summary

Controllable Multi-Interest Framework for Recommendation

Yukuo Cen, Jianwei Zhang, Xu Zou, Chang Zhou, Hongxia Yang, Jie Tang

arXiv:2005.09347v2May 19, 2020cs.IRcs.LGstat.ML

TL;DR

Sequential recommender systems often represent a user’s behavior history with one embedding, which cannot reflect multiple interests. ComiRec extracts multiple interests for candidate retrieval and aggregates the resulting items with controllable accuracy–diversity behavior. The framework improves over state-of-the-art models on real-world datasets and has been deployed on Alibaba’s distributed cloud platform.

Problem
A unified user embedding cannot reflect a user’s multiple interests during a period, limiting representation for sequential recommendation.
Method
ComiRec extracts multiple interests from user behavior sequences, retrieves candidate items for each interest, and aggregates them into an overall recommendation with controllable accuracy and diversity.
Results
ComiRec achieves significant improvements over state-of-the-art models on two real-world sequential-recommendation datasets and reports recall@50 gains of 1.39% and 8.65% over MIND for ComiRec-SA and ComiRec-DR.
Takeaways & Limitations
The framework supports multi-interest candidate retrieval and overall recommendation in an industrial matching setting, with reported deployment on Alibaba’s distributed cloud platform.

Abstract

from arXiv · show

Recently, neural networks have been widely used in e-commerce recommender systems, owing to the rapid development of deep learning. We formalize the recommender system as a sequential recommendation problem, intending to predict the next items that the user might be interacted with. Recent works usually give an overall embedding from a user's behavior sequence. However, a unified user embedding cannot reflect the user's multiple interests during a period. In this paper, we propose a novel controllable multi-interest framework for the sequential recommendation, called ComiRec. Our multi-interest module captures multiple interests from user behavior sequences, which can be exploited for retrieving candidate items from the large-scale item pool. These items are then fed into an aggregation module to obtain the overall recommendation. The aggregation module leverages a controllable factor to balance the recommendation accuracy and diversity. We conduct experiments for the sequential recommendation on two real-world datasets, Amazon and Taobao. Experimental results demonstrate that our framework achieves significant improvements over state-of-the-art models. Our framework has also been successfully deployed on the offline Alibaba distributed cloud platform.

1 INTRODUCTION

Sequential recommendation predicts users’ next interactions, but a single embedding cannot represent multiple concurrent interests. ComiRec addresses this with multi-interest extraction, candidate retrieval, and aggregation for overall recommendations.

Sequential recommendation predicts the next item a user may be interested in from their behavior history.
A unified user embedding cannot adequately represent multiple interests expressed during the same period.Emma’s click sequence illustrates distinct interests in jewelry, handbags, and make-ups.
ComiRec captures multiple user interests from behavior sequences and exploits them to retrieve candidate items.The framework is designed for sequential recommendation and uses an interest-based retrieval stage.
An aggregation module combines items retrieved from different interests to produce the overall top-N recommendation.Figure 1 illustrates independent retrieval for each interest followed by aggregation.
The paper reports state-of-the-art performance on two real-world datasets and successful deployment on Alibaba’s distributed cloud platform.The contributions identify Amazon and Taobao as challenging sequential-recommendation datasets.

2 RELATED WORK

Related work spans classical, neural, sequential, and diversity-oriented recommender systems, alongside attention and capsule-network methods. These studies motivate modeling sequential behavior, multiple interests, and recommendation diversity.

Classical recommender systems include collaborative filtering, matrix factorization, and factorization machines for modeling user-item interactions.These methods differ in how they identify similarities or represent interactions in latent spaces.
Neural recommender models extend interaction modeling with nonlinear networks and explicit low- or high-order feature interactions.Examples include NCF, NFM, DeepFM, xDeepFM, and DMF.
Sequential recommendation methods model ordered user behavior, using Markov chains, recurrent networks, local activation, or attention-based mechanisms.Examples include FPMC, HRM, GRU4Rec, DIN, and SDM.
Recommendation-diversity research distinguishes aggregated diversity from the dissimilarity of items recommended to an individual user.The literature notes that maximizing accuracy alone can yield similar and boring recommendations.
Attention mechanisms and capsule networks have been adapted to recommendation, with MIND using dynamic routing to extract diverse user interests.These techniques provide methodological context for interest representation in recommender systems.

3 METHODOLOGY

ComiRec formulates sequential recommendation around multi-interest user representations for large-scale matching, then aggregates interest-specific candidates with controllable accuracy–diversity trade-offs.

Problem Formulation: Sequential recommendation predicts the next item from a user’s time-ordered historical interactions.Industrial systems separate matching, which retrieves candidates, from ranking, which sorts them using more precise scores.
Problem Formulation: The matching stage is crucial because industrial item pools contain millions or billions of items, making user-embedding quality decisive for candidate retrieval.ComiRec focuses primarily on improving matching effectiveness using fast KNN retrieval.
Multi-Interest Framework: ComiRec transforms item IDs into embeddings and extracts multiple user interests instead of relying on a single embedding.The framework explores dynamic routing and self-attention as alternative multi-interest extraction methods.
Aggregation Module: Each interest retrieves top-N candidates, and a controllable greedy aggregation selects N items by balancing relevance and diversity through λ.λ = 0 gives the most accurate case, while λ = ∞ gives the most diverse case.
Multi-Interest Framework: Dynamic routing treats sequence item embeddings as primary capsules and user interests as interest capsules, producing an interest matrix through iterative coupling and squashing.The resulting matrix Vu contains K interest embeddings for downstream training and serving.
Multi-Interest Framework: Self-attention produces behavior weights and combines them into interest representations, with repeated attention extended to multiple interests and positional embeddings added for sequence order.The attention vector weights user behaviors, while the multi-interest extension forms an attention matrix.

4 EXPERIMENTS

The experiments evaluate ComiRec on public sequential-recommendation datasets and a billion-scale industrial dataset, examining accuracy, parameter sensitivity, controllability, and practical deployment.

Experimental Setup: The framework is evaluated under strong generalization by training on training users and predicting held-out behaviors for validation and test users.Users are split 8:1:1, and the remaining 20% of validation and test behaviors are predicted during evaluation.
Experimental Setup: The evaluation uses two public datasets, Amazon and Taobao, with sequential recommendation experiments and an additional billion-scale industrial dataset.Amazon uses Books reviews and metadata with sequences truncated at 20; Taobao uses time-sorted click behaviors truncated at 50.
Quantitative Results: ComiRec retrieves top-N candidates independently for each of K interests, then selects the final top-N items from K·N candidates by inner-product ranking.The same retrieval procedure is applied to MIND, while λ is set to 0 for fair comparison.
Quantitative Results: ComiRec models outperform all state-of-the-art baselines by a wide margin on every reported evaluation criterion, with ComiRec-DR outperforming MIND more than ComiRec-SA.ComiRec-SA obtains comparable results with ComiRec-DR, while GRU4Rec performs best among single-embedding models.
Quantitative Results: The best number of interests depends on both model and dataset: ComiRec-SA performs best at K = 2 or 6 on Amazon and K = 2 on Taobao, while ComiRec-DR performs best at K = 4 on Amazon.On Taobao, ComiRec-DR improves as K increases from 2 to 8.
Controllable Study: Increasing λ substantially increases recommendation diversity while slightly decreasing recall, enabling an accuracy–diversity trade-off through hyperparameter selection.The controllable aggregation module can achieve an optimum trade-off by choosing an appropriate λ.
Industrial Results: On the industrial dataset, ComiRec-SA and ComiRec-DR improve recall@50 over MIND by 1.39% and 8.65%, respectively.The industrial dataset contains 22 million items, 145 million users, and 4 billion behaviors.
Industrial Results: A case study shows that four learned interests correspond approximately to item categories and retrieve similar items without manually defined category information.The model is trained using item IDs and still recovers categories from user behavior sequences.

5 CONCLUSION

The paper concludes that ComiRec provides controllable multi-interest sequential recommendation with strong results on public datasets and effectiveness and efficiency confirmed in industrial use.

5 CONCLUSION: ComiRec generates multiple user interests and aggregates their retrieved items into an overall top-N recommendation.The framework integrates multi-interest extraction with controllable aggregation.
5 CONCLUSION: Experiments show significant improvements over state-of-the-art models on two challenging datasets, with deployment on Alibaba’s distributed cloud platform.Results on a billion-scale industrial dataset further confirm effectiveness and efficiency in practice.

A APPENDIX

The appendix provides implementation notes for the proposed models, followed by details of other models and dataset descriptions.

A APPENDIX: The appendix contains implementation notes for the proposed models and then describes other models and datasets.It serves as a supporting reference for experiment and model details.

A.1 Implementation Notes

The implementation uses separate environments for public-dataset experiments and the industrial dataset, with distributed Alibaba infrastructure for the latter.

A.1 Implementation Notes: Public-dataset experiments run on one Linux server with four Intel Xeon CPUs, 256G RAM, and eight NVIDIA RTX 2080 Ti GPUs.These experiments use TensorFlow 1.14 and Python 3.6.
A.1 Implementation Notes: Industrial experiments run on Alibaba’s distributed cloud platform containing thousands of workers, with two workers sharing one NVIDIA Tesla P100 GPU.The industrial implementation uses TensorFlow 1.4 and Python 2.7.
A.1 Implementation Notes: Training samples are formed by selecting a random item as the label and using the preceding click items as the training sequence.The implementation separates data iteration, model training, and evaluation.
A.1 Implementation Notes: Embeddings use dimension 64, sampled softmax uses 10 samples, training runs for at most 1 million iterations, and early stopping uses validation Recall@50.The batch sizes are 128 for Amazon and 256 for Taobao; dynamic routing uses three iterations.
A.1 Implementation Notes: The code and the authors’ partitions of the two public datasets are available.This supports reproduction of the public-dataset experiments.

A.2 Compared Methods

The compared-method implementations train on user click sequences through multi-interest extraction, sampled softmax loss, and Adam updates. The comparison includes established baselines such as MostPopular, YouTube DNN, and GRU4REC.

Training procedure: The training procedure initializes parameters, generates samples from user click sequences, and iterates over batches until convergence.Each batch proceeds through multi-interest extraction, computation based on Equation (8), sampled softmax loss, and parameter updates.
Training procedure: The model computes user representations with a multi-interest extraction module before calculating the sampled softmax loss.The procedure computes V_u through multi-interest extraction and then computes v_u based on Equation (8).
Training procedure: Adam is used to update the model parameters.
Compared methods: MostPopular recommends the most popular items without personalization or training.It is implemented separately from the trained models.
Compared methods: The comparison includes YouTube DNN and GRU4REC as deep-learning and recurrent-neural-network recommendation baselines.YouTube DNN is implemented from its original paper, while GRU4REC uses GRUCell and dynamic_rnn implementations.

A.3 Datasets

The experiments use two public datasets and a billion-scale industrial dataset, retaining users and items with at least five behaviors in the public datasets. Amazon9 uses time-ordered Books reviews to predict subsequent review behavior.

Dataset overview: The evaluation covers three datasets: two public datasets and one billion-scale industrial dataset.
Dataset overview: For the two public datasets, users and items must have at least 5 behaviors.
Amazon9: Amazon9 uses Amazon Books reviews and metadata, ordering each user’s reviews by time to predict later review behavior.Training samples are truncated at length 20.