PaperlayerView exact revision on arXiv

Source-linked AI summary

Product-based Neural Networks for User Response Prediction over Multi-field Categorical Data

Yanru Qu, Bohui Fang, Weinan Zhang, Ruiming Tang, Minzhe Niu, Huifeng Guo, Yong Yu, Xiuqiang He

arXiv:1807.00311v1cs.IRcs.LGstat.ML

TL;DR

User response prediction over multi-field categorical data is difficult because one-hot inputs are sparse and existing latent-vector and DNN models have identified gradient problems. The paper introduces kernel products and Product-based Neural Networks to learn feature interactions, achieving consistent benchmark gains and a 34.67% relative CTR improvement for PIN online.

  • Problem

    Sparse one-hot representations and coupled or insensitive gradients limit learning for multi-field categorical user response prediction.

  • Method

    The paper proposes kernel products for field-aware interactions and Product-based Neural Networks with product feature extractors and DNN classifiers.

  • Results

    34.67% relative CTR improvement was achieved by PIN in an online A/B test, while the models consistently outperformed 8 baselines on AUC and log loss.

  • Takeaways & Limitations

    Combining expressive product-based feature extractors with powerful DNN classifiers yields state-of-the-art performance on 4 industrial datasets.

  • Takeaways & Limitations

    CCPM’s convolutional design fails to model interactions among non-neighbor fields, and DNN gradient sensitivity decreases for large collections of uncorrelated functions.

Abstract

from arXiv · show

User response prediction is a crucial component for personalized information retrieval and filtering scenarios, such as recommender system and web search. The data in user response prediction is mostly in a multi-field categorical format and transformed into sparse representations via one-hot encoding. Due to the sparsity problems in representation and optimization, most research focuses on feature engineering and shallow modeling. Recently, deep neural networks have attracted research attention on such a problem for their high capacity and end-to-end training scheme. In this paper, we study user response prediction in the scenario of click prediction. We first analyze a coupled gradient issue in latent vector-based models and propose kernel product to learn field-aware feature interactions. Then we discuss an insensitive gradient issue in DNN-based models and propose Product-based Neural Network (PNN) which adopts a feature extractor to explore feature interactions. Generalizing the kernel product to a net-in-net architecture, we further propose Product-network In Network (PIN) which can generalize previous models. Extensive experiments on 4 industrial datasets and 1 contest dataset demonstrate that our models consistently outperform 8 baselines on both AUC and log loss. Besides, PIN makes great CTR improvement (relatively 34.67%) in online A/B test.

1 INTRODUCTION

User response prediction over sparse, multi-field categorical data supports personalized retrieval and filtering, but existing models face representation and optimization challenges. The paper analyzes these issues and proposes product-based models that outperform established baselines.

  • User response prediction estimates responses such as likes, clicks, or purchases for personalized information retrieval, advertising, recommender systems, and web search.
  • Multi-field categorical instances are one-hot encoded into high-dimensional sparse binary vectors, creating a central modeling challenge.
  • FM uses shared latent vectors across interaction types, producing a coupled gradient issue that can reduce model capacity.
  • PNN combines an embedding layer, product-based feature extractor, and DNN classifier to model feature interactions.FM, KFM, and NIFM yield IPNN, KPNN, and PIN variants.
  • 34.67% relative CTR improvement was achieved by PIN in an online A/B test, while the models consistently outperformed 8 baselines on AUC and log loss.The evaluation covered 4 large-scale real-world datasets and 1 contest dataset.

2 BACKGROUND AND RELATED WORK

Prior work spans linear, latent-vector, tree, and DNN-based models for sparse categorical response prediction. These approaches differ in feature extraction, interaction modeling, optimization, and reliance on feature engineering.

  • User response prediction is commonly treated as binary classification with cross-entropy or log loss, and evaluated using AUC, log loss, and relative information gain.
  • LR extracts linear information, FM extracts bi-linear information, and GBDT explores feature combinations non-parametrically.
  • Latent-vector models represent categorical data compactly, while FFM assigns multiple latent vectors to each category for field-aware interactions.
  • DNN-based approaches include FNN, NFM, AFM, and related models that use embeddings or neural layers to model recommendation interactions.
  • CCPM models neighbor-field convolutions but fails to model convolutions among non-neighbor fields, while WDL requires manual feature engineering and DeepFM replaces its wide component with FM.
  • Network In Network introduces micro neural networks between convolutional layers to abstract data within receptive fields and model local dependencies.

3 METHODOLOGY

The methodology decomposes learning multi-field categorical data into representation and classification, then uses product operations to improve feature extraction before DNN classification.

  • The paper separates the difficulty of learning multi-field categorical data into representation and classification phases.
  • The methodology first studies field-aware feature interactions, then analyzes DNN classifier deficiencies, and finally presents Product-based Neural Networks.
  • Cross entropy, or log loss, is used as the training objective for user response prediction.The label y is binary, and the model outputs the predicted probability of a positive response.

3.1 Field-aware Feature Interactions

The section develops field-aware feature interactions to address coupled gradients in latent vector models, using kernel products and their neural generalizations. These constructions preserve manageable space complexity while expanding interaction capacity.

  • Multi-field response records select one category per field, enabling feature combinations such as “Male and London and Tuesday implies True”.
  • FM couples different inter-field interactions because one latent vector is reused, updating v_Male toward both v_London and v_Tue.The resulting coupled gradient can degrade model capacity when fields should interact differently.
  • Field-aware latent vectors decouple interaction gradients, but FFM requires O(Nnk) parameters and therefore restricts latent-vector size.
  • Kernel product defines interactions as p^Tϕq, generalizing vector inner and outer products through a learnable projection matrix.The kernel-product form uses k^2 + k multiplications and additions, compared with 2k^2 multiplications and k^2 additions for the outer-product convolution form.
  • Projecting latent vectors into field-specific kernel spaces relaxes FFM while retaining O(Nk) total space complexity because O(n^2k^2) kernel parameters are much smaller than Nk.
  • NIFM uses a micro network for each field pair, producing nonlinear feature interactions and allowing interaction complexity to be controlled.KFM and NIFM use field-aware kernels, whereas AFM uses a field-sharing attention network.

3.2 Training Feature Interactions with Trees or DNNs is Difficult

The section argues that sparse multi-field categorical targets can contain high-order and periodic feature interactions that trees and gradient-trained DNNs struggle to learn. It motivates product layers as explicit feature extractors for DNNs.

  • Parity-check targets accept inputs with an odd number of matched conditions and reject inputs with an even number, revealing periodic feature-interaction structure.The examples include three matched conditions accepted and two or zero matched conditions rejected.
  • Basic feature combinations can induce parity checks, so a positive feature set does not imply that its subsets or supersets are positive.
  • For a large collection of uncorrelated target functions, DNN gradient variance decreases linearly with the hypothesis-space size |H|.The resulting gradient becomes concentrated around a point independent of the target function, reducing useful optimization information.
  • Because gradient-based DNNs have difficulty learning feature interactions implicitly, the paper proposes product layers to provide explicit interaction extraction.

3.3 Product-based Neural Networks

PNN combines field-wise embeddings, explicit product-based interaction extraction, and a DNN classifier. Its IPNN, KPNN, and PIN variants use different interaction extractors, with PIN extending the architecture through micro networks.

  • Architecture: PNN separates modeling into embedding, interaction, and DNN-classifier components, using the interaction layer to extract feature interactions.The classifier receives embeddings and extracted interactions as inputs.
  • Model variants: IPNN uses inner products, KPNN uses kernel products, and PIN uses micro networks as feature extractors.These variants correspond respectively to FM, KFM, and NIFM feature extractors.
  • Model variants: Unlike Neural FM, IPNN concatenates pairwise interactions before passing them to the DNN classifier instead of summing them.This preserves separate interaction values as classifier inputs.
  • Product-network In Network: PIN merges several micro-network forward passes into a single tensor multiplication and can use vector-valued sub-network outputs.Fused layer normalization is used to stabilize the small sub-networks.
  • Product-network In Network: PIN processes concatenated embedding pairs through micro networks, while removing redundant direct embedding-to-DNN connections.Each sub-network receives an embedding pair and a product term, with normalization optionally inserted into hidden layers.

4 PRACTICAL ISSUES

The paper examines practical difficulties caused by sparse categorical inputs, including initialization, adaptive optimization, regularization, and normalization. It describes gradient instability and distributional bias that can complicate training.

  • Data processing: Categorical and numerical inputs have different distributions and comparability, so numerical fields are converted to categorical representations or set embeddings are averaged.Bucketing or clustering assigns numerical values to categorical buckets.
  • Initialization: Sparse, field-wise embedding connections differ from dense fully connected layers, making standard initialization assumptions less direct.The paper compares random initialization with pre-training and discusses initialization scales for sparse embeddings.
  • Optimization: The logit gradient decays exponentially early in training, causing gradients of parameters that depend on it to decrease dramatically.This behavior is shown for typical models using SGD or Adam.
  • Optimization: Adam’s estimated gradient can saturate across part of the real-gradient domain, and its threshold increases with ε and training step t.The paper concludes that ε affects convergence and training stability, especially on unbalanced datasets.
  • Optimization: Long-tailed adaptive gradients may cause training instability or parameter divergence on sparse inputs, motivating sparse updates.Sparse updates apply estimated gradients only to parameters involved in forward propagation.
  • Regularization and normalization: A small mini-batch can have large distributional bias in sparse data, while dropout may amplify that bias.Batch normalization can also fail on sparse input because it relies on mini-batch statistics, although larger batches may be impractical.

5 EXPERIMENTS

The experiments compare the proposed models with eight baselines across four datasets using AUC and log loss. Kernel-product models improve on latent-vector baselines, while PNNs consistently outperform DeepFM and PIN ranks best across all datasets.

  • Experimental setup: The evaluation compares models on four datasets using AUC and log loss, with controlled embedding sizes and DNN structures for fair comparison.The datasets are Criteo, Avazu, iPinYou, and Huawei.
  • Experimental setup: The study compares eight baselines: LR, GBDT, FM, FFM, FNN, CCPM, AFM, and DeepFM.Implementations use XGBoost, libFFM, TensorFlow, and MXNet.
  • Latent-vector models: KFM and NIFM outperform FFM and AFM on all four datasets, supporting the effectiveness of kernel-product methods.FFM is the strongest baseline on Criteo, Avazu, and Huawei, while AFM is strongest on iPinYou.
  • DNN-based models: PNNs consistently outperform DeepFM, and PIN achieves the best results on all datasets.The paper treats these results as evidence for the effectiveness of product layers.

4 datasets.

The parameter study evaluates adaptive embeddings, model dimensions, and network structure across four datasets, while an online test compares PIN with FTRL.

  • Adaptive embedding: Adaptive embeddings further improve KFM and NIFM, but are harder to parallelize and train much slower.Embedding size is set from each field’s category count using ki = min(c log(Ni), K).
  • Embedding size: FM and DeepFM are easier to overfit with large embedding sizes.The study tests multiple embedding sizes on Criteo, Avazu, iPinYou, and Huawei.
  • Network structure: PIN consistently outperforms FNN and DeepFM across tested network widths and depths.PIN also performs better when networks are small, indicating more expressive feature representations.
  • Network structure: When networks are large, FNN and DeepFM are easier to overfit, whereas PIN is more robust.Network width and depth are varied after fixing embedding sizes and other parameters according to the study procedure.
  • Online A/B test: 34.67% relative CTR improvement was observed for PIN over FTRL after a 14-day Huawei App Market A/B test.The maximum and minimum relative improvements were 63.26% and 14.72%, respectively.

5.2 Practical Issues

The practical-issues study examines memory, training speed, initialization, optimization sensitivity, regularization, normalization, and activation choices for these models.

  • Space and time complexity: PNNs require more memory than FNN because their extra feature extractors add parameters.Among compared models, CCPM uses the least memory among DNN-based models through convolutional parameter sharing.
  • Space and time complexity: Kernel products have O(n^2k^2) complexity, making KFM and KPNN relatively slow when embedding vectors are large.A Criteo Challenge trick reduces kernel-product complexity to O(n^2k).
  • Initialization: FM pre-training does not always outperform random initialization on FNN and AFM.The authors conclude that the preferable embedding initialization depends on the dataset.
  • Activation functions: ReLU performs best among the tested activation functions on FNN for Avazu.The authors suggest more efficient gradient propagation as a possible reason.
  • Regularization and normalization: Dropout decreases AUC on Avazu, especially with small batch sizes, while layer normalization stabilizes dropout across batch sizes.The reported dropout behavior depends on data sparsity.

5.3 Feature Interaction Visualization

The visualization study represents field-level interactions with heatmaps and finds that kernel products produce sparser, clearer interaction patterns than FM and FFM.

  • Visualization method: Heatmap cells encode inner or kernel products between mean embeddings of pairs of Avazu fields.Both axes represent the 24 fields, and diagonal cells are set to zero because fields do not interact with themselves.
  • Visualization method: Bright or dark isolated points indicate large-magnitude field interactions, whereas bright or dark bars indicate broad interactions across fields.The visualization expects isolated points under the sparse-interaction assumption.
  • Results: FM shows coupled field patterns, FFM produces shorter but still unwanted bars, and KFM displays mostly isolated points.The authors interpret this pattern as evidence that kernel products solve the coupled gradient issue.
  • Results: Most useful interactions occur among fields 1–13, while fields 14–24 provide less interactive information.The authors suggest using more complex models for the former fields and potentially compressing the latter.

5.4 Training Difficulty of Gradient-based DNN

Synthetic experiments test how sparsity and model capacity affect gradient-based DNN training on data generated from a poly-2 function.

  • Data sparsity: DNN training becomes more difficult as the input data becomes sparser.Sparsity is varied by changing field sizes while fitting a three-hidden-layer DNN.
  • DNN fitting: DNNs remain below the poly-2 performance upper bound, and increasing width or depth does not close the gap.The experiment uses one- and three-hidden-layer networks with varying widths and additional depth settings.
  • DNN fitting: The results indicate that gradient-based DNNs cannot perfectly fit a simple poly-2 function despite universal approximation.The authors use this finding to motivate extracting feature interactions from sparse inputs before DNN classification.

6 CONCLUSION

The paper addresses gradient problems in latent vector- and DNN-based models for multi-field categorical user response prediction. Its PNN models outperform eight baselines across four industrial datasets, while PIN improves CTR in an online A/B test.

  • Kernel product methods address coupled gradients in latent vector-based models by learning field-aware feature interactions and alleviating FFM’s memory bottleneck.
  • PNNs address insensitive gradients in DNN-based models by combining expressive feature extractors with DNN classifiers.
  • PNNs consistently outperform 8 baselines and achieve state-of-the-art performance on 4 industrial datasets.
  • PIN makes great CTR improvements in an online A/B test.
Loading 1807.00311v1…