PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep & Cross Network for Ad Click Predictions

Ruoxi Wang, Bin Fu, Gang Fu, Mingliang Wang

arXiv:1708.05123v1cs.LGstat.ML

TL;DR

Feature engineering for predictive cross features is difficult, while DNNs learn interactions implicitly and may be inefficient for some bounded-degree interactions. DCN adds an explicit cross network to a DNN, and experiments report superior accuracy and memory usage across CTR and dense classification settings.

  • Problem

    Predictive cross-feature discovery often requires manual engineering or exhaustive search, while DNNs may inefficiently learn some interactions implicitly.

  • Method

    DCN jointly trains a DNN with a cross network that explicitly learns bounded-degree feature interactions by crossing features at successive layers.

  • Results

    DCN outperforms other models on the Criteo CTR dataset and achieves the best test accuracy of 0.9740 on forest covertype data with the least memory consumption.

  • Takeaways & Limitations

    DCN provides automatic, explicit bounded-degree feature learning with lower logloss than a DNN using nearly an order of magnitude fewer parameters.

  • Takeaways & Limitations

    The authors identify deeper cross-network training, polynomial-approximation efficiency, and cross–deep-network optimization as areas requiring further investigation.

Abstract

from arXiv · show

Feature engineering has been the key to the success of many prediction models. However, the process is non-trivial and often requires manual feature engineering or exhaustive searching. DNNs are able to automatically learn feature interactions; however, they generate all the interactions implicitly, and are not necessarily efficient in learning all types of cross features. In this paper, we propose the Deep & Cross Network (DCN) which keeps the benefits of a DNN model, and beyond that, it introduces a novel cross network that is more efficient in learning certain bounded-degree feature interactions. In particular, DCN explicitly applies feature crossing at each layer, requires no manual feature engineering, and adds negligible extra complexity to the DNN model. Our experimental results have demonstrated its superiority over the state-of-art algorithms on the CTR prediction dataset and dense classification dataset, in terms of both model accuracy and memory usage.

1 INTRODUCTION

CTR prediction matters because publisher revenue depends heavily on accurate predictions, yet sparse categorical data makes feature exploration difficult and cross-feature discovery often requires manual engineering. DCN addresses this by jointly training an explicit cross network with a DNN to learn bounded-degree interactions automatically and efficiently.

  • Accurate CTR prediction is economically important because publisher revenue relies heavily on predicting whether users click ads.
  • Web-scale recommender data is largely discrete and categorical, creating a large sparse feature space that complicates discovering predictive and rare cross features.
  • Linear models scale easily but have limited expressive power, while identifying useful cross features typically requires manual engineering or exhaustive search.
  • DCN introduces a cross network that explicitly applies feature crossing automatically and trains it jointly with a DNN.
  • Cross-network interaction degree is determined by layer depth, with each layer adding higher-order interactions while retaining those learned earlier.
  • DCN combines automatic bounded-degree cross-feature learning with sparse and dense inputs, avoiding manual engineering and exhaustive searching at low computational cost.

2 DEEP & CROSS NETWORK (DCN)

DCN embeds and stacks sparse and dense inputs, then processes them through parallel cross and deep networks whose outputs are combined for classification. The cross network explicitly builds bounded-degree feature interactions while retaining prior interactions, with linear complexity and joint training with the deep network.

  • 2.1 Embedding and Stacking Layer: DCN embeds sparse categorical features, stacks them with normalized dense features, and feeds the resulting vector into parallel cross and deep networks.Embeddings reduce the dimensionality of one-hot categorical inputs before stacking.
  • 2.2 Cross Network: Each cross layer applies feature crossing, adds its input back, and learns the residual mapping between successive cross-layer outputs.The layer uses a crossing function f and parameters w_l and b_l.
  • 2.2 Cross Network: An l-layer cross network has highest polynomial degree l + 1 and contains cross terms of degrees 1 through l + 1.Interaction degree is therefore controlled by cross-network depth.
  • 2.2 Cross Network: The cross network has linear time and space complexity in input dimension, adding negligible complexity relative to the deep network.This efficiency uses the rank-one property of x_0x_0^T to generate cross terms without storing the full matrix.
  • 2.3 Deep Network: The deep network is a fully connected feed-forward network, while the combination layer concatenates cross and deep outputs and feeds them to a logits layer.For two-class classification, the logits layer uses the sigmoid function.
  • 2.4 Combination Layer: DCN jointly trains the cross and deep networks so that each network is aware of the other during training.The combined model is optimized with log loss and an L2 regularization term.

3 CROSS NETWORK ANALYSIS

The cross network is analyzed as a layered polynomial approximator that explicitly represents bounded-degree feature interactions while sharing parameters efficiently. The analysis connects its structure to factorization machines and shows how efficient projection avoids the cost of explicitly materializing pairwise interactions.

  • 3.1 Polynomial Approximation: The cross network approximates the polynomial class of the same degree, with efficiency, expressiveness, and improved generalization to real-world datasets.The analysis uses polynomial approximation under smoothness assumptions as its theoretical perspective.
  • 3.1 Polynomial Approximation: With only O(d) parameters, the cross network contains all cross terms of a same-degree polynomial while assigning distinct coefficients to those terms.The polynomial class itself has O(d^n) coefficients, highlighting the network’s parameter efficiency.
  • 3.2 Generalization of FMs: Parameter sharing makes the model more efficient and supports generalization to unseen feature interactions, including sparse-feature settings where co-occurring feature pairs are rare.The corresponding cross-term weights combine parameters associated with the individual features.
  • 3.2 Generalization of FMs: The cross network extends factorization-machine parameter sharing from a single layer to multiple layers and high-degree cross-terms.Unlike shallow FMs, its highest interaction degree is bounded by layer depth, while its parameter count grows linearly with input dimension.
  • 3.3 Efficient Projection: Each cross layer implicitly forms pairwise interactions between the input and current representation, then projects them back to input dimension with cost linear in d.A direct construction of all d^2 interactions would incur cubic cost, whereas the cross layer uses an equivalent compact computation.

4 EXPERIMENTAL RESULTS

Experiments compare DCN with DNN, logistic regression, factorization machines, and deep crossing on CTR and dense classification data. DCN generally achieves better predictive performance while using substantially less memory, with cross-layer depth affecting validation logloss.

  • Model performance: DCN outperforms the compared models on Criteo test logloss and uses only 40% of DNN memory.The comparison includes DNN, logistic regression, factorization machines, and deep crossing.
  • Model performance: 0.4422 ± 9 × 10^-5 test logloss is reported for DCN, versus 0.4430 ± 3.7 × 10^-4 for DNN and 0.4430 ± 4.3 × 10^-4 for DC.These values are means and standard deviations across 10 independent runs.
  • Memory efficiency: Nearly an order of magnitude fewer parameters is needed by DCN than by a single DNN to reach desired logloss thresholds.The paper attributes this efficiency to bounded-degree feature interactions learned by the cross network.
  • Memory efficiency: DCN consistently outperforms DNN under fixed memory budgets, although DNN closes part of the gap in the large-parameter regime.The remaining advantage suggests efficient learning of meaningful interactions that a large DNN may not learn as effectively.
  • Cross-layer effects: Adding one cross layer clearly improves validation logloss, while further layers can decrease, fluctuate, or slightly increase it depending on the setting.Decreases indicate useful cross terms; increases indicate that some higher-degree interactions are unhelpful.
  • Non-CTR datasets: DCN achieves test accuracy 0.9740 on forest covertype and test logloss 0.4494 on Higgs, compared with 0.9737 and 0.4506 for DNN, respectively.On Higgs, DCN uses half the memory of DNN; forest covertype also reports the least memory consumption for DCN.

5 CONCLUSION AND FUTURE DIRECTIONS

The paper concludes that DCN addresses inefficiencies in implicit deep feature learning by explicitly learning bounded-degree cross features alongside deep representations. It reports superiority over state-of-the-art methods on sparse and dense datasets in accuracy and memory usage, while identifying several directions for further study.

  • Conclusion: DCN avoids manual feature crafting and exhaustive search while targeting feature interactions that DNNs may learn inefficiently.The conclusion contrasts DCN’s explicit crossing with DNNs’ implicit, highly nonlinear feature learning.
  • Conclusion: DCN learns explicit bounded-degree cross features jointly with traditional deep representations for sparse and dense inputs.The highest cross-feature degree increases by one at each cross layer.
  • Conclusion: Experiments demonstrate superiority over state-of-the-art algorithms on sparse and dense datasets in model accuracy and memory usage.
  • Future directions: Future work includes deeper cross-network training, polynomial-approximation analysis, and studying cross-network interactions with deep networks during optimization.

Appendix: Proof of Teorem 3.1

The appendix proves the theorem by induction over cross-network layers and then derives polynomial coefficients by collecting terms associated with feature permutations.

  • Notation: The multi-index notation represents monomials formed from feature coordinates, with the final index entry fixed at 1.The proof uses this convention when relating successive layer expressions.
  • Coefficient derivation: Rearranging the expanded expression yields the desired theorem claim after the coefficient contributions are collected.
  • Induction proof: The proof establishes the claimed cross-network expression by induction on the layer index.The induction step expands the expression from layer k to layer k + 1 and uses the fixed final index entry.
  • Induction proof: The induction step derives the next-layer term by replacing occurrences of the previous layer’s weight with the new layer weight.This substitution connects the polynomial form at consecutive cross layers.
  • Coefficient derivation: The coefficient of each monomial is obtained by summing weights associated with all corresponding feature permutations.The proof defines active indices and permutation sets to collect these contributions.
Loading 1708.05123v1…