DCN V2: Improved Deep & Cross Network and Practical Lessons for Web-scale Learning to Rank Systems

Ruoxi Wang; Rakesh Shivanna; Derek Z. Cheng; Sagar Jain; Dong Lin; Lichan Hong; Ed H. Chi

Source-linked AI summary

DCN V2: Improved Deep & Cross Network and Practical Lessons for Web-scale Learning to Rank Systems

Ruoxi Wang, Rakesh Shivanna, Derek Z. Cheng, Sagar Jain, Dong Lin, Lichan Hong, Ed H. Chi

arXiv:2008.13535v2Aug 19, 2020cs.IRcs.LGstat.ML

TL;DR

Web-scale ranking requires effective feature crosses, but manual search is expensive and DNN-based approaches can be inefficient, while DCN has limited expressiveness. The paper proposes DCN-V2, including low-rank and mixture-of-low-rank variants, to improve explicit interaction modeling while retaining efficiency and simplicity. Experiments and production deployments report stronger benchmark performance and significant offline and online gains.

Problem
Sparse web-scale feature spaces make manual cross search exhaustive, while DNNs can inefficiently model feature crosses and DCN’s cross network has limited expressiveness.
Method
DCN-V2 combines expressive cross layers for explicit interactions with a deep network for implicit interactions, and uses low-rank mixtures for performance-latency trade-offs.
Results
DCN-V2 significantly outperforms SOTA methods on benchmark datasets and delivers significant offline accuracy and online business metric gains in web-scale ranking systems.
Takeaways & Limitations
DCN-V2 provides a simple, efficient building block for learning explicit and implicit feature interactions in large-scale learning-to-rank systems.
Takeaways & Limitations
The paper leaves investigation of how the imposed low-rank structure affects whole-system training dynamics and global statistics for future work.

Abstract

from arXiv · show

Learning effective feature crosses is the key behind building recommender systems. However, the sparse and large feature space requires exhaustive search to identify effective crosses. Deep & Cross Network (DCN) was proposed to automatically and efficiently learn bounded-degree predictive feature interactions. Unfortunately, in models that serve web-scale traffic with billions of training examples, DCN showed limited expressiveness in its cross network at learning more predictive feature interactions. Despite significant research progress made, many deep learning models in production still rely on traditional feed-forward neural networks to learn feature crosses inefficiently. In light of the pros/cons of DCN and existing feature interaction learning approaches, we propose an improved framework DCN-V2 to make DCN more practical in large-scale industrial settings. In a comprehensive experimental study with extensive hyper-parameter search and model tuning, we observed that DCN-V2 approaches outperform all the state-of-the-art algorithms on popular benchmark datasets. The improved DCN-V2 is more expressive yet remains cost efficient at feature interaction learning, especially when coupled with a mixture of low-rank architecture. DCN-V2 is simple, can be easily adopted as building blocks, and has delivered significant offline accuracy and online business metrics gains across many web-scale learning to rank systems at Google.

1 INTRODUCTION

Web-scale learning to rank needs efficient feature-interaction learning because manual crossing is combinatorial, while larger neural networks create serving and training costs. DCN-V2 addresses DCN’s expressiveness limits with a more expressive, efficient architecture and low-rank extensions.

Manual feature-cross search is combinatorial, exhaustive, domain-dependent, and difficult to generalize in sparse web-scale applications.
DNNs can improve ranking performance but inefficiently approximate even 2nd- or 3rd-order feature crosses.
Wider or deeper networks can improve feature-cross modeling while increasing serving latency and trainability problems under high-QPS constraints.
DCN’s cross network efficiently learns bounded-degree interactions but has limited expressiveness because its polynomial class uses O(input size) parameters.
DCN-V2 learns explicit interactions through cross layers and combines them with a deep network for complementary implicit interactions.
DCN-V2 adds more expressive feature-cross learning while remaining efficient and simple, with low-rank and mixture-of-experts techniques supporting performance-latency trade-offs.

2 RELATED WORK

Prior feature-interaction models combine explicit and implicit interactions through parallel or stacked architectures, but they face feature-engineering, expressiveness, or computational-cost limitations. DCN remains a strong baseline, motivating a more expressive architecture that preserves its simple structure.

Recent feature-interaction methods combine explicit multiplicative operations or learned pairwise functions with implicit interactions from DNNs.
Parallel Structure: Parallel wide-and-deep models use raw-feature crosses in the wide component and a DNN in the deep component, but selecting wide crosses remains feature engineering.
Parallel Structure: DeepFM automates wide-component interactions with factorization machines, while DCN learns explicit bounded-degree interactions automatically and efficiently.
Parallel Structure: xDeepFM increases DCN’s expressiveness with multiple feature maps but has significantly high computational cost, reported as 10x of #params.
Stacked Structure: Stacked architectures place interaction layers between embeddings and DNNs, while product-based networks can incur high computational cost.
DCN remains a strong baseline because of its simple optimization-friendly structure, although limited expressiveness restricts its ability to learn more effective crosses.

3 PROPOSED ARCHITECTURE: DCN-V2

DCN-V2 combines explicit cross layers with a deep network, increasing DCN’s expressiveness while supporting arbitrary embedding sizes and cost-efficient low-rank approximations. Cross layers model bounded-degree interactions, while stacked or parallel combinations integrate explicit and implicit interactions.

DCN-V2 combines an embedding layer, explicit cross network, and deep network to learn explicit and implicit feature interactions.
3.1 Embedding Layer: The embedding layer concatenates embedded categorical features with normalized dense features into x0.
3.1 Embedding Layer: The model supports arbitrary embedding sizes, unlike related methods requiring equal embedding dimensions, which suits industrial recommenders with varying vocabulary sizes.
3.2 Cross Network: DCN-V2 cross layers compute x_l+1 = x0 ⊙(W_lx_l + b_l) + x_l, with an l-layer network containing all crosses up to polynomial order l + 1.
3.4 Deep and Cross Combination: Stacked and parallel structures combine cross and deep networks, with the better architecture depending on the data.
3.5 Cost-Effective Mixture of Low-Rank DCN: DCN-Mix uses multiple low-rank experts and input-dependent gating, reducing cross-network complexity to O(2drKL_c) when rK ≪ d.

4 MODEL ANALYSIS

The analysis characterizes DCN-V2’s polynomial interaction capacity and relates its feature-interaction component to established methods. An l-layer cross network creates feature interactions through order l + 1, with DCN-V2 offering a more expressive parameterization than DCN.

The analysis treats individual vector elements and whole feature embeddings as separate units for characterizing polynomial interactions.
An l-layer cross network creates all feature interactions up to order l + 1 from both bitwise and feature-wise perspectives.
DCN-V2 characterizes the same polynomial class as DCN-V but with more parameters and greater expressiveness, including feature-wise interactions.
Connections to Related Methods: FM’s pairwise inner-product interactions are equivalent to one DCN-V2 layer without the residual term under a structured weight matrix.
Connections to Related Methods: The first xDeepFM feature map is equivalent to a one-layer DCN-V2 without the residual term.
Connections to Related Methods: AutoInt and DCN-V2 share a high-level layered interaction structure, but differ in how feature interactions are modeled.
Connections to Related Methods: Unlike PNN’s explicit construction of all pairwise interactions, DCN-V2 implicitly creates them using a structured matrix.

5 RESEARCH QUESTIONS

The paper evaluates when feature-interaction methods are more efficient than ReLU-based DNNs, how baselines perform independently, and whether mDCN improves accuracy–cost trade-offs and interpretability. It also studies how mDCN settings affect quality and whether the model captures important crosses.

RQ1 asks when feature-interaction learning methods become more efficient than ReLU-based DNNs.
RQ2 asks how each baseline’s feature-interaction component performs without integration with a DNN.
RQ3 compares mDCN with baselines and examines whether mDCN and mixture-of-low-rank DCN improve the accuracy–cost trade-off.
RQ4 studies how mDCN settings affect model quality.
RQ5 examines whether mDCN captures important feature crosses and provides model understandability.
“CrossNet” or “CN” denotes the cross network, while “Mix” denotes its mixture-of-low-rank version.

6 EMPIRICAL UNDERSTANDING OF FEATURE CROSSING TECHNIQUES (RQ1)

The experiments compare ReLU networks, DCN, and DCN-V2 on synthetic polynomial-fitting tasks with controlled cross complexity and layer depth. They show that DCN-V2 remains accurate on complicated interactions and is more stable as redundant higher-order crosses are introduced.

Performance with increasing difficulty: Synthetic datasets vary feature-cross difficulty through sparsity, cross count, and similarity of cross patterns.The experiments use monomial interactions and randomly assigned inputs and weights to create datasets with increasing difficulty.
Performance with increasing difficulty: DCN-V2 remains accurate as cross patterns become more complicated, while DCN degrades and DNN remains poor even with wider, deeper structures.The comparison includes f1 through f3, with DNN layer sizes expanded for the more difficult setting.
Role of each component: A cross term x0 ⊙(Wx_i) models order-d crosses at layer d-1, so order-3 fitting performs best at layer 2, with similar behavior for order 4.The layer-depth experiment uses homogeneous polynomials of orders 3 and 4.
Role of each component: Bias and residual terms preserve crosses up to the highest order, reducing degradation from redundant interactions and stabilizing performance across layers.This stabilization is especially relevant when real-world cross patterns are unknown.
Combined-order polynomial fitting: For combined-order polynomial fitting, CN-M captures higher-order crosses as depth increases without performance degradation beyond layer 3.Table 2 reports mean RMSE over 5 runs; the stability is attributed to bias and residual terms.
Performance with increasing difficulty: ReLUs are inefficient for explicit multiplicative feature crosses, even with deeper and larger networks.The study reports that accuracy degrades further as cross patterns become more complicated.

7 EXPERIMENTAL RESULTS (RQ2 - RQ5)

Experiments across multiple datasets and platforms compare DCN-V2 with tuned baselines, finding strong quality, efficiency, trainability, and feature-crossing behavior. Analyses also identify practical limits involving model depth, rank, mixture-of-experts gating, and replacing ReLU layers.

Model Quality: DCN-V2 models outperformed all baselines, with stacked preferred on Criteo and parallel preferred on MovieLens-1M.Both structures outperformed the baselines in the reported optimal settings.
Model Quality: DCN-V2 achieved the best performance by explicitly modeling up to 3rd-order crosses beyond DNN-learned implicit interactions.DCN-Mix reduced cost by 30% while maintaining accuracy.
Trainability: PNN and xDeepFM showed trainability problems through high variance or standard deviation across datasets and trials.PNN’s Criteo mean LogLoss was driven up by its high standard deviation, while xDeepFM showed high variance on MovieLens-1M.
Model Quality: DCN-V2 consistently outperformed DNN, which the authors associate with balanced parameter allocation and the cross network’s simple structure.The simple cross-network structure also eased optimization.
Model Efficiency: DCN-Mix reduced cost by 30% while maintaining accuracy, while DCN-V2 delivered the best performance with relatively efficient computation.The comparison was based on tuned model size and FLOPS settings.
Cross Layers and ReLU: Replacing ReLU layers entirely with cross layers remains a preliminary hypothesis requiring substantially more analysis and experiments.The observation motivates future exploration rather than establishing a general replacement strategy.
Hyper-parameters: Deeper cross networks improved quality, but gains slowed with depth; beyond two layers, the cross network could close and surpass DNN performance.The results suggest lower-order crosses contributed more than higher-order crosses.
Hyper-parameters: Increasing rank from 4 to 64 nearly linearly decreased LogLoss, while further increases slowed improvement, identifying 64 as a threshold rank on Criteo.The authors hypothesize that important cross signals were captured in the top-64 singular values.

8 PRODUCTIONIZING DCN-V2 AT GOOGLE

DCN-V2 was productionized in a large-scale Google recommender system using hundreds of billions of training examples and sparse features ranging from 2 to millions of vocabulary entries. Compared with a production model, it improved AUCLoss by 0.6% and produced significant online gains, while practical results favored inserting and stacking a small number of cross layers.

Hundreds of billions of training examples and sparse-feature vocabularies ranging from 2 to millions characterize the production setting.
Comparisons with Production Models: 0.6% AUCLoss (1 - AUC) improvement was achieved by DCN-V2 compared with the production model.A 0.1% AUCLoss gain is considered significant for this model.
Comparisons with Production Models: DCN-V2 also produced significant online performance gains on key business metrics.
Practical Learnings: Cross layers work best when inserted between the input and hidden DNN layers, where feature representations and interactions retain stronger physical meaning.
Practical Learnings: Accuracy gains were consistent with stacking or concatenating 1 - 2 cross layers, while gains began to plateau beyond two layers.

9 CONCLUSIONS AND FUTURE WORK

The paper concludes that DCN-V2 provides expressive and simple explicit-cross modeling, while DCN-Mix improves the performance-latency trade-off through low-rank structure. It reports successful deployment and identifies several directions for extending understanding and improving the gating mechanism.

Conclusions: DCN-V2 models explicit feature crosses expressively while retaining a simple structure.
Conclusions: DCN-Mix uses the low-rank nature of the cross-network weight matrix to improve the trade-off between model performance and latency.
Conclusions: DCN-V2 has been deployed in multiple web-scale learning-to-rank systems with significant offline accuracy and online business-metric gains.
Future Work: Future work includes studying optimization interactions, the relation between embeddings and matrix rank, improved DCN-Mix gating, and broader architectures such as RNNs and CNNs.

10 BASELINE PERFORMANCE REPORTED IN PAPERS

This section lists Logloss and AUC metrics quoted from papers for each baseline, with rows representing baselines and columns representing reporting papers.

Logloss and AUC metrics are quoted from papers for each baseline.

11 THEOREM PROOFS

The theorem-proof section establishes the cross-network interaction formula by induction and characterizes the feature interactions represented by successive cross layers. It also develops notation for embeddings, index sets, permutations, and multi-index monomials used in the proof.

Proof Strategy: The proof proceeds by establishing notations and then proving the result by induction.
Notation: The embedding is represented as x = [x1; x2; . . . ; x𝑐], with each feature embedding x𝑖 and cross-layer output x𝑙𝑖 defined in their respective embedding spaces.
Notation: The sets S𝑖p represent combinations of p feature indices with replacement whose first element is fixed to i.
Proof Strategy: The induction proof separates combinations according to whether they include the new index k + 1, enabling the recursive equality used in the theorem.
Result: For l ≥ 1, the l-th cross layer contains all feature-wise interactions of order up to l + 1.
Special Case: The scalar-embedding case treats each input element as a unit and assumes the final logit is 1⊤x𝑙 to simplify the resulting formula.
Derivation: The final derivation rewrites cross terms as monomials indexed by α and combines the summations using permutation and combination index sets.