Source-linked AI summary
Learning Tree-based Deep Model for Recommender Systems
Han Zhu, Xiang Li, Pengye Zhang, Guozheng Li, Jie He, Han Li, Kun Gai
TL;DR
Large-corpus recommenders struggle to retrieve candidates when expressive models require costly user–item scoring. TDM uses top-down hierarchical retrieval with jointly learned trees, and experiments report significant gains across datasets plus effectiveness in Taobao production, while noting scope limitations in hierarchical-softmax comparison and preliminary online deployment results.
Problem
Full-corpus prediction is computationally difficult, while inner-product retrieval limits the use of more expressive models and candidate novelty.
Method
TDM traverses a learned item tree from coarse to fine, using neural discriminators to rank promising nodes and retrieve top-k leaves.
Results
TDM significantly outperforms existing methods on two large-scale datasets, including 21.1% and 42.6% recall improvements over YouTube product-DNN, and shows effectiveness in Taobao online evaluation.
Takeaways & Limitations
Tree-based retrieval makes arbitrary advanced models feasible for large-corpus recommendation while exploring the entire corpus and supporting novel, effective results.
Takeaways & Limitations
Hierarchical softmax is unsuitable for large-scale retrieval, and the reported online improvement is preliminary because deployment covered only the first TDM DNN version.
Abstract
from arXiv · showhide
Model-based methods for recommender systems have been studied extensively in recent years. In systems with large corpus, however, the calculation cost for the learnt model to predict all user-item preferences is tremendous, which makes full corpus retrieval extremely difficult. To overcome the calculation barriers, models such as matrix factorization resort to inner product form (i.e., model user-item preference as the inner product of user, item latent factors) and indexes to facilitate efficient approximate k-nearest neighbor searches. However, it still remains challenging to incorporate more expressive interaction forms between user and item features, e.g., interactions through deep neural networks, because of the calculation cost. In this paper, we focus on the problem of introducing arbitrary advanced models to recommender systems with large corpus. We propose a novel tree-based method which can provide logarithmic complexity w.r.t. corpus size even with more expressive models such as deep neural networks. Our main idea is to predict user interests from coarse to fine by traversing tree nodes in a top-down fashion and making decisions for each user-node pair. We also show that the tree structure can be jointly learnt towards better compatibility with users' interest distribution and hence facilitate both training and prediction. Experimental evaluations with two large-scale real-world datasets show that the proposed method significantly outperforms traditional methods. Online A/B test results in Taobao display advertising platform also demonstrate the effectiveness of the proposed method in production environments.
1 INTRODUCTION
Large-corpus recommendation must balance prediction cost, expressive user–item interactions, and novel candidate discovery. TDM addresses these challenges with hierarchical tree search and jointly learned item hierarchies.
- Motivation: Linear prediction over enormous corpora makes full-corpus recommendation computationally unacceptable.This creates a deployment challenge for otherwise well-performing recommendation algorithms.
- Motivation: Recommendation quality requires novelty as well as precision, so results should not consist only of items resembling users’ historical behaviors.
- Motivation: Inner-product models support efficient approximate kNN retrieval but limit more expressive interactions between user and item representations.Neural networks, cross-product features, and attention-based models offer richer interactions but cannot directly use this retrieval form.
- Proposed approach: TDM converts recommendation into hierarchical classification, predicting user interests from coarse concepts to fine-grained items through a tree.The approach leverages item hierarchies such as smartphone and iPhone to divide a large problem into easier successive decisions.
- Proposed approach: TDM jointly learns the tree structure and neural network toward an item hierarchy compatible with users’ interests, improving retrieval and training.
- Related work: TDM differs from hierarchical softmax because retrieval requires exploring top-k nodes layer-wise rather than following only each level’s most probable path.Hierarchical softmax’s binary discrimination can also lose the capacity to represent interest in both similar child nodes.
2 SYSTEM ARCHITECTURE
Taobao’s advertising recommender first retrieves a much smaller candidate set from a corpus of hundreds of millions of items, then applies more expressive models and ranking. TDM operates in the retrieval stage, where efficiency and candidate quality jointly constrain impressions.
- Candidate retrieval: The matching server uses user, context, and item features to shrink hundreds of millions of corpus items to usually hundreds of candidates.The tree-based model reduces the candidate-set size by several orders of magnitude.
- System role: The architecture therefore places TDM in the matching stage, before computationally heavier prediction and ranking models.
- Downstream serving: After retrieval, real-time prediction models estimate indicators such as click-through rate or conversion rate before strategic ranking and impression.
- Candidate retrieval: Candidate generation is essential because user interest in the retrieved items sets an upper bound on impression quality.
3 TREE-BASED DEEP MODEL
TDM uses a learned recommendation tree and hierarchical node-probability modeling to retrieve preferred items from large corpora with logarithmic prediction cost. Its max-heap-like formulation, level-wise training, and top-down search are designed to support expressive models while improving retrieval quality.
- Tree Structure: A recommendation tree maps each corpus item to one leaf, while non-leaf nodes represent coarse-grained concepts.Each non-root node has one parent and an arbitrary number of children.
- Related Work: Hierarchical softmax is unsuitable for large-scale retrieval because greedy top-down choices can miss better leaves and local binary decisions may lose global discrimination.The formulation still requires traversing the corpus to identify the most probable leaf and can propagate poor upper-level decisions.
- Tree-based Model Formulation: TDM replaces hierarchical softmax with a max-heap-like tree whose parent preference equals the maximum child preference after layer-specific normalization.This ordering supports retrieving high-preference nodes layer by layer without requiring exact probabilities for every tree node.
- Hierarchical Retrieval: For top-k retrieval, TDM explores only children of layer-wise top-k nodes in a top-down search, traversing at most 2*k*log |C| nodes in a complete binary tree.The logarithmic relation to corpus size makes advanced binary probability models feasible, while hierarchical refinement may improve recommendation quality over brute-force search.
- Tree-based Model Formulation: Training uses each interacted leaf and its ancestors as positive samples, with randomly selected same-level nodes as negatives for level-specific order discrimination.The sampling strategy makes each level’s discriminator an intra-level global one rather than only distinguishing optimal from suboptimal child choices.
- Tree-based Model Formulation: The binary probability model takes user state and a sampled node as input, and its outputs are optimized with a positive-and-negative-sample likelihood loss.Tree-node embeddings and the tree structure are also learned as parts of the model; profile and contextual features can be added.
- Tree Construction and Learning: Tree construction may use expert knowledge or item concurrence and similarity, but clustered trees can become imbalanced and harm training and retrieval.The paper identifies tree construction as a practical constraint when domain-specific expert knowledge is unavailable.
4 ONLINE SERVING
The online serving system separates asynchronous user-feature assembly from tree-based candidate retrieval. Real-time behavior updates prepare features before page-view requests trigger layer-wise neural retrieval.
- Online Serving: User features are assembled asynchronously from behaviors such as clicks, purchases, and cart additions, while page-view requests trigger candidate retrieval.The real-time feature server updates inputs, and the user targeting server uses pre-assembled features.
- Online Serving: During serving, the trained neural network calculates node-preference probabilities as retrieval proceeds layer by layer.The system follows the top-down retrieval procedure described by the model’s prediction algorithm.
5 EXPERIMENTAL STUDY
Experiments on MovieLens-20M and Taobao UserBehavior evaluate TDM against established recommendation methods, including novelty-focused settings and tree/model variants. TDM attention-DNN outperforms baselines across datasets and improves online advertising metrics, while learned trees and hierarchical search further support performance and efficiency.
- Evaluation: The evaluation compares Precision@M, Recall@M, F-Measure@M, and Novelty@M across FM, BPR-MF, item-CF, YouTube product-DNN, and TDM variants.Metrics are averaged over testing users, and reported values for methods with variance are averaged across five runs.
- Comparison Results: 21.1% and 42.6% recall improvements over YouTube product-DNN are reported for TDM attention-DNN on the two datasets without filtering.TDM attention-DNN significantly outperforms all baselines on most metrics in both datasets; item-CF has the weakest novelty results.
- Comparison Results: TDM attention-DNN also outperforms all baselines by a large margin after previously interacted items are filtered from recommendation results.The recommendation and ground-truth sets are complemented to the required size when filtering reduces their cardinality.
- Comparison Results: 34.3% better recall than YouTube product-DNN is achieved when interacted categories are excluded in UserBehavior, while item-CF recall is 1.06%.The category-level setting evaluates exploration beyond users’ historically interacted categories.
- Empirical Analysis: The attention module yields near 10% recall improvement in UserBehavior, and hierarchical-softmax substitution performs much worse than TDM attention-DNN.The comparisons indicate that neural interaction forms and attention improve TDM performance, whereas hierarchical softmax is not well matched to this recommendation problem.
- Empirical Analysis: Learned tree structures improve training and prediction, increasing filtered-category recall from 4.15% to 4.82% and surpassing YouTube product-DNN’s 3.09%.The learned tree also produces smaller test loss and better convergence than the initial tree.
- Online Results: In Taobao advertising, TDM increases CTR by 2.1% and RPM by 6.4%, while deployed TDM DNN recommendation averages about 6 milliseconds per request.The reported prediction complexity is O(k ∗ log |C| ∗ t), where k is the result size, |C| the corpus size, and t the network feed-forward complexity.
6 CONCLUSION
The paper identifies prediction computation as the main challenge for large-scale model-based recommendation and proposes tree-based retrieval with jointly learned structure. Experiments validate TDM’s accuracy, novelty, and production effectiveness.
- 6 CONCLUSION: TDM addresses the calculation problem in large-corpus prediction with a tree-based coarse-to-fine approach that supports advanced models.The method traverses user interests along a tree and uses tree structure learning to improve compatibility with interest distributions.
- 6 CONCLUSION: TDM is validated on recommendation accuracy and novelty through extensive experiments and empirical analysis.The paper reports that the experiments validate effectiveness in both dimensions.
- 6 CONCLUSION: TDM was deployed in Taobao display advertising production, where it improved business benefits and user experience.