Source-linked AI summary
Multi-Interest Network with Dynamic Routing for Recommendation at Tmall
Chao Li, Zhiyuan Liu, Mengmeng Wu, Yuchi Xu, Pipei Huang, Huan Zhao, Guoliang Kang, Qiwei Chen, Wei Li, Dik Lun Lee
TL;DR
Industrial recommendation must retrieve and rank items at billion-scale while representing users whose interests span multiple categories. MIND learns multiple user-interest vectors using dynamic routing and label-aware attention for matching, and it reports superior benchmark performance plus improved homepage CTR after Tmall deployment.
Problem
Billion-scale matching requires user representations that capture diverse interests, whereas existing deep models commonly represent each user with one vector.
Method
MIND uses dynamic routing to cluster historical behaviors into multiple interest representations and label-aware attention to train item-specific user representations for matching.
Results
MIND achieves superior performance on public benchmarks and an industrial Tmall dataset, while its deployed system significantly improves homepage CTR.
Takeaways & Limitations
MIND provides a deployed multi-vector approach for representing diverse user interests in Tmall’s matching-stage recommendation system.
Abstract
from arXiv · showhide
Industrial recommender systems usually consist of the matching stage and the ranking stage, in order to handle the billion-scale of users and items. The matching stage retrieves candidate items relevant to user interests, while the ranking stage sorts candidate items by user interests. Thus, the most critical ability is to model and represent user interests for either stage. Most of the existing deep learning-based models represent one user as a single vector which is insufficient to capture the varying nature of user's interests. In this paper, we approach this problem from a different view, to represent one user with multiple vectors encoding the different aspects of the user's interests. We propose the Multi-Interest Network with Dynamic routing (MIND) for dealing with user's diverse interests in the matching stage. Specifically, we design a multi-interest extractor layer based on capsule routing mechanism, which is applicable for clustering historical behaviors and extracting diverse interests. Furthermore, we develop a technique named label-aware attention to help learn a user representation with multiple vectors. Through extensive experiments on several public benchmarks and one large-scale industrial dataset from Tmall, we demonstrate that MIND can achieve superior performance than state-of-the-art methods for recommendation. Currently, MIND has been deployed for handling major online traffic at the homepage on Mobile Tmall App.
1 INTRODUCTION
Tmall’s billion-scale recommendation system must model diverse user interests efficiently across matching and ranking. MIND addresses this challenge by representing users with multiple interest vectors and reports improved online CTR after deployment.
- Recommendation Setting: Tmall’s recommendation process uses matching to retrieve relevant candidates and ranking to predict interaction probabilities for those candidates.The matching stage retrieves thousands of items from a billion-scale pool.
- Motivation: Users interact with products across different categories, so capturing diverse interests is vital for Tmall recommendation.The paper notes that the same user may be interested in various item categories.
- MIND: MIND uses dynamic routing to softly cluster historical behaviors into multiple user representation vectors, each corresponding to a particular interest.The vectors are computed once and reused in matching to retrieve relevant items from billion-scale inventories.
- Contributions: MIND combines a multi-interest extractor with label-aware attention and shows superior performance on public datasets and a Tmall industrial dataset.The system also includes an online-serving pipeline for billion-scale users.
- Deployment: MIND significantly improves homepage click-through rate after deployment on the Mobile Tmall App.The homepage accounts for about half of Tmall’s total traffic.
2 RELATED WORK
Prior recommendation work represents users through interacted items, latent factors, or neural embeddings, while capsule networks provide a routing-based alternative for encoding structured representations.
- Deep Learning for Recommendation: Deep recommendation models commonly use neural networks to learn low-dimensional user embeddings from behavior or other input representations.The related work discusses neural approaches for user and item interaction modeling.
- User Representation: Collaborative filtering represents user interests with historical items or hidden factors, but these approaches face sparsity or computational demands.These limitations motivate alternative user-representation methods.
- Capsule Network: Capsule networks use dynamic routing to learn connection weights between capsules, distinguishing them from conventional neural networks.The paper introduces capsules as groups of neurons that output whole vectors.
3 METHOD
MIND formalizes matching as retrieving relevant items from a billion-scale pool using multiple user vectors and item embeddings. These representations are learned from behavior, profile, and item features for efficient candidate retrieval.
- Problem Formalization: The matching stage retrieves a small relevant subset from a billion-scale item pool for each user.The target is to reduce the pool to thousands of candidate items.
- MIND Overview: MIND’s overview combines embedded and pooled inputs with multi-interest extraction, profile features, ReLU transformations, and label-aware training before approximate-nearest-neighbor retrieval.Serving uses multiple user representation vectors for item lookup.
- Problem Formalization: Each training instance contains interacted items, user profile features, and target-item features.Examples include user gender and age, plus target item and category identifiers.
- Problem Formalization: MIND maps raw features into K user representation vectors, with K = 1 corresponding to the single-vector YouTube DNN setting.The user representation has dimensionality d × K.
- Problem Formalization: Item representations are learned through an embedding function, and candidate items are retrieved by scoring user and item representations.The matching stage returns a predefined number N of items.
3.2 Embedding & Pooling Layer
The embedding and pooling layer converts high-dimensional categorical identifiers into dense vectors and averages item embeddings before later interest extraction.
- Embedding: MIND embeds extremely high-dimensional categorical identifiers into low-dimensional dense vectors to reduce parameters and ease learning.The input includes profile, behavior, and target-item feature groups.
- Pooling: Embeddings of each behavior item are averaged by a pooling layer before the behavior representations enter later processing.Profile feature embeddings are concatenated separately.
3.3 Multi-Interest Extractor Layer
MIND represents a user with multiple interest vectors rather than compressing diverse interests into one vector. Its multi-interest extractor uses Behavior-to-Interest dynamic routing to cluster historical behaviors into interest capsules, with design choices for shared spaces, initialization, and adaptive capsule counts.
- Multiple representation vectors separate diverse user interests for more accurate matching-stage item retrieval.
- The multi-interest extractor clusters historical behaviors into groups and infers one interest representation vector for each group.
- B2I Dynamic Routing: Behavior-to-Interest dynamic routing adapts capsule routing to aggregate behavior capsules into interest representation vectors.
- Shared bilinear mapping matrix: A shared bilinear mapping matrix keeps interest capsules in the same vector space and supports variable-length user behavior sequences.
- Randomly initialized routing logits: Randomly initialized routing logits make initial interest capsules different, avoiding identical capsules across routing iterations.
- Dynamic interest number: The number of interest capsules is adjusted per user, saving computing and memory resources for users with fewer interests.
3.4 Label-aware Attention Layer
MIND uses label-aware attention to select the interest capsule most relevant to a target item and form an item-specific user representation. A tunable exponent controls attention concentration, with hard attention yielding faster convergence in the experiments.
- Label-aware attention lets each target item select the relevant interest capsule from the multiple capsules generated by the extractor.
- The label item embedding acts as the query, while interest capsules serve as both keys and values in the attention mechanism.
- The exponent p controls attention concentration: values near 0 distribute attention evenly, whereas larger p emphasizes larger dot products.
- Hard attention in the experiments leads to faster convergence.
3.5 Training & Serving
During training, MIND computes interaction probabilities and an overall objective over user-item interactions, but the billion-item denominator is computationally prohibitive. Sampled softmax makes training tractable, while serving maps each user to multiple vectors for approximate-nearest-neighbor retrieval.
- The training objective is defined over user-item interaction data, with user and label-item vectors used to compute interaction probabilities.
- The denominator’s sum is computationally prohibitive because the item population scales to billions.
- Table 1 reports statistics for the two datasets used in offline evaluation.
- Sampled softmax makes the objective function tractable for training MIND.
- At serving time, user behavior and profile inputs produce multiple representation vectors, which retrieve top N items using approximate nearest neighbors.
3.6 Connections with Existing Methods
MIND shares deep-neural-network-based retrieval with YouTube DNN but uses multiple user vectors, and it shares DIN’s goal of modeling diverse interests while differing in mechanism and stage applicability.
- YouTube DNN: YouTube DNN represents each user with one vector, whereas MIND uses multiple vectors for large-scale matching-stage retrieval.
- YouTube DNN: When K = 1, MIND degenerates to YouTube DNN, making MIND a generalization of that model.
- DIN: MIND and DIN both model diverse interests, but DIN uses item-level attention while MIND generates interest capsules through dynamic routing.
- DIN: DIN targets the ranking stage for thousands of items, whereas MIND is designed for the matching stage.
4 EXPERIMENTS
Offline and online experiments evaluate MIND across public and Tmall datasets, showing that multiple dynamically extracted interests improve recommendation quality and support practical large-scale serving.
- Comparing Methods: Methods using multiple user representation vectors generally outperform WALS, YouTube DNN, and MIND-1-interest on both datasets.The results support multiple vectors as an effective way to model diverse interests and improve recommendation accuracy.
- Comparing Methods: MIND’s dynamic routing extracts multiple interests through clustering, while label-aware attention matches target items against those interest vectors.These components are reported to produce more precise user representations and more accurate user-item matching.
- Analysis of Hyperparameters: MIND is robust to different σ values, while larger p values improve performance compared with p = 0.The authors choose σ = 1 for practical applications; p = 0 gives equal attention to every interest without reference to the label.
- Online Experiments: MIND with 5 ~7 interests achieves the best online CTR and significantly beats item-based CF and YouTube DNN.Performance improves as interests increase from 1 to 5, then remains nearly constant; dynamic interest numbers have comparable performance to 7 interests and can reduce serving cost.
- Interpretability Analysis: Coupling coefficients show that behavior classes align with corresponding interest capsules, while recalled items are more strongly associated with MIND interests than with YouTube DNN.The visualizations also show different interest granularity across users.
5 SYSTEM DEPLOYMENT
MIND is deployed within Tmall’s recommendation workflow, where User Interest Extractor and Recall Engine retrieve candidates before ranking. The system retrieves candidates from a billion-scale item pool in under 15 milliseconds and ranks the top 1000 by predicted CTR.
- Deployment workflow: Figure 7 presents the architecture of the recommendation system at Tmall.
- Deployment workflow: Tmall’s deployment workflow sends recommendation requests to the Tmall Personality Platform, which retrieves recent user behaviors for the User Interest Extractor.The extracted interests support candidate retrieval before final recommendation results are assembled.
- Deployment workflow: The deployment workflow separates candidate retrieval from ranking, with retrieved candidates passed to the ranking stage for final recommendation.
- Serving performance: Candidate selection from the billion-scale item pool by the User Interest Extractor and Recall Engine takes less than 15 milliseconds.This reflects a tradeoff between item-pool scope and system response time.
- Serving performance: The Ranking Service scores the top 1000 candidates using features to predict CTRs, after which the platform completes the recommendation list.
6 CONCLUSION AND FUTURE WORK
The paper proposes MIND to represent diverse user interests for matching in billion-scale e-commerce recommendation. It reports superior offline performance on public benchmarks and online CTR results demonstrating effectiveness and feasibility at Tmall, while identifying richer behavior information and improved routing initialization as future directions.
- Conclusion: MIND represents users’ diverse interests for the matching stage in e-commerce recommendation involving billion-scale users and items.
- Method: MIND uses a multi-interest extractor with variant dynamic routing and trains the extracted interests with label-aware attention.
- Results: Offline experiments show superior performance on public benchmarks, while online CTRs demonstrate MIND’s effectiveness and feasibility at Tmall’s live production.
- Future work: Future work will incorporate additional behavior-sequence information, such as behavior time, and improve dynamic-routing initialization using a K-means++-related scheme.