PaperlayerView exact revision on arXiv

Source-linked AI summary

Feature Generation by Convolutional Neural Network for Click-Through Rate Prediction

Bin Liu, Ruiming Tang, Yingzhi Chen, Jinkai Yu, Huifeng Guo, Yuzhou Zhang

arXiv:1904.04447v1cs.IRcs.LG

TL;DR

CTR models struggle to learn sparse useful interactions, while manual feature engineering is costly and domain-dependent. FGCNN automatically generates features with CNN and MLP before classification, and it outperforms nine state-of-the-art models across three large-scale datasets while supporting other classifiers. The paper concludes that automatic identification of important features can reduce DNN learning difficulties.

  • Problem

    Sparse useful interactions are difficult for DNNs to learn with many parameters, while manually engineered features are costly and require domain knowledge.

  • Method

    FGCNN combines CNN-based local-pattern extraction with MLP recombination to generate features, then applies a Deep Classifier to the augmented feature space.

  • Results

    FGCNN outperforms nine state-of-the-art models on three large-scale datasets, and other models used as Deep Classifiers also achieve better performance.

  • Takeaways & Limitations

    Automatically identifying important features is presented as a useful direction for reducing the learning difficulties of deep CTR models.

  • Takeaways & Limitations

    Generating too many features can add noise, increase MLP learning difficulties, and reduce performance.

Abstract

from arXiv · show

Click-Through Rate prediction is an important task in recommender systems, which aims to estimate the probability of a user to click on a given item. Recently, many deep models have been proposed to learn low-order and high-order feature interactions from original features. However, since useful interactions are always sparse, it is difficult for DNN to learn them effectively under a large number of parameters. In real scenarios, artificial features are able to improve the performance of deep models (such as Wide & Deep Learning), but feature engineering is expensive and requires domain knowledge, making it impractical in different scenarios. Therefore, it is necessary to augment feature space automatically. In this paper, We propose a novel Feature Generation by Convolutional Neural Network (FGCNN) model with two components: Feature Generation and Deep Classifier. Feature Generation leverages the strength of CNN to generate local patterns and recombine them to generate new features. Deep Classifier adopts the structure of IPNN to learn interactions from the augmented feature space. Experimental results on three large-scale datasets show that FGCNN significantly outperforms nine state-of-the-art models. Moreover, when applying some state-of-the-art models as Deep Classifier, better performance is always achieved, showing the great compatibility of our FGCNN model. This work explores a novel direction for CTR predictions: it is quite useful to reduce the learning difficulties of DNN by automatically identifying important features.

1 INTRODUCTION

CTR prediction estimates click probabilities and is central to recommendation revenue, but sparse useful interactions make deep models difficult to optimize. FGCNN automatically generates features with CNN and MLP, then uses them with raw features for classification.

  • Motivation: CTR prediction estimates whether a user will click an item and affects advertising revenue through CTR×bid ranking.In online advertising, the system ranks candidate advertisements by CTR multiplied by the profit received after a click.
  • Motivation: Feature engineering can improve deep-model performance but is expensive and requires domain knowledge, motivating automatic feature generation.Wide & Deep Learning is cited as improving offline AUC by 0.6% and online CTR by 1% with artificial features.
  • Motivation: Sparse useful feature interactions make it difficult for deep models to learn effectively from large parameter spaces.A one-hidden-layer MLP may need irrelevant feature weights, such as those for Name and Height, to become exactly zero.
  • Proposed approach: FGCNN combines CNN and MLP to identify local patterns, recombine them into global new features, and augment the raw feature space.The model contains Feature Generation and Deep Classifier components; the classifier can adopt existing state-of-the-art network structures.
  • Results: Experiments on three large-scale datasets show that FGCNN outperforms nine state-of-the-art models and remains compatible with alternative deep classifiers.Using other models as Deep Classifiers also improves performance, while the CNN+MLP structure is reported as more robust to raw-feature order than traditional CNN.

2.1 Overview

FGCNN has Feature Generation and Deep Classifier components. Feature Generation identifies useful local and global patterns to create features that complement raw features, while the classifier learns from the augmented space.

  • 2.1 Overview: FGCNN consists of Feature Generation and Deep Classifier components.The overview figure presents the model architecture and its best Avazu-Dataset hyper-parameter setting.
  • 2.1 Overview: Feature Generation identifies useful local and global patterns to generate new features as a complement to raw features.The resulting augmented feature space combines raw and generated features before classification.
  • 2.1 Overview: Deep Classifier learns and predicts from the augmented feature space through a deep learning model.The paper describes this component as the prediction stage following feature generation.

2.2 Feature Embedding

FGCNN represents categorical fields with low-dimensional embeddings. These embeddings form a matrix used by Feature Generation and Deep Classifier, with a separate classifier embedding matrix to avoid inconsistent gradient directions.

  • 2.2 Feature Embedding: CTR instances are commonly represented as high-dimensional sparse binary vectors after one-hot encoding multi-field categorical data.Numerical features are usually converted to categorical form by bucketing.
  • 2.2 Feature Embedding: An embedding layer compresses raw features into low-dimensional vectors, summing feature embeddings for multivalent fields.For a univalent field, the field embedding is its feature embedding; for a multivalent field, embeddings are summed.
  • 2.2 Feature Embedding: Each field embedding e_i is represented as a k-dimensional vector, and the instance embeddings form matrix E ∈ R^n_f × k.The supplied notation passages define field embeddings and the resulting embedding matrix.
  • 2.2 Feature Embedding: FGCNN uses the embedding matrix in both Feature Generation and Deep Classifier, while a separate E′ is used for the classifier to avoid inconsistent gradient directions.E is used for Feature Generation and E′ for Deep Classifier.

2.3 Feature Generation

Feature Generation combines CNN and MLP-based recombination to identify sparse local patterns, capture global feature interactions, and generate features that augment the raw feature space.

  • Feature Generation: CNN learns neighbor feature patterns with fewer parameters, while recombination combines them into global feature interactions.The recombination layer is fully connected and addresses global non-neighbor interactions that CNN alone can miss.
  • Convolutional Layer: Each instance is reshaped into an embedding tensor before convolutional layers extract neighbor feature interactions.The first convolutional layer produces feature maps from the reshaped embedding matrix.
  • Pooling Layer: Max-pooling selects important feature interactions, reduces parameters, and supplies pooled outputs to subsequent convolutional layers.The pooling output becomes the input to the next convolutional layer.
  • Recombination Layer: A fully connected recombination layer transforms pooled local patterns into new features because CNN outputs can omit global non-neighbor interactions.The layer generates nf/h_i p_i m_i^r features at each recombination stage, as specified in the section’s notation.
  • Feature Augmentation: Multiple CNN-plus-recombination rounds generate feature sets that are aggregated and concatenated with raw features for CTR prediction.The resulting augmented embedding matrix is used by the Deep Classifier.

2.4 Deep Classifier

The Deep Classifier uses IPNN to model interactions in the augmented feature space, combining explicit pairwise interactions with MLP-based implicit interaction learning.

  • Network Structure: IPNN is used as the Deep Classifier because it offers a trade-off between model complexity and accuracy.The paper notes that IPNN is more efficient than PIN while achieving slightly lower performance in prior evaluation.
  • FM Layer: The FM layer computes pairwise interactions among the augmented embedding fields using inner products.The interaction vector contains inner products between pairs of field embeddings.
  • MLP: IPNN concatenates the FM interaction results with the augmented embedding matrix and feeds them into an MLP.The MLP learns implicit feature interactions through successive hidden layers.
  • Training: Batch Normalization is applied before each activation function to accelerate FGCNN training.The normalization uses mini-batch activation statistics and learnable scale and shift parameters.
  • Objective Function: FGCNN minimizes cross entropy between binary labels and predicted click probabilities.The label y is binary, while the predicted probability lies between zero and one.

2.5 Complexity Analysis

FGCNN’s complexity is determined by feature embedding, feature generation, and the deep classifier, with costs dominated by feature counts, hidden layers, embedding size, and convolutional parameters.

  • FGCNN space complexity combines Feature Embedding, Feature Generation, and Deep Classifier components.
  • Feature Generation produces T = nf + Ínc i=1 Ni feature embeddings after combining raw and generated features.
  • Deep Classifier’s first hidden layer receives pairwise products and embeddings of raw and generated features.
  • The overall time complexity is O(N1k Ínc 1k^2 + T^2H1 + Íq i=2 HiHi−1), dominated by generated features, hidden neurons, embedding size, and convolutional parameters.

3 EXPERIMENTS

The experiments evaluate FGCNN against state-of-the-art CTR models across three datasets, using AUC and log loss with repeated trials and significance testing.

  • The experiments address performance, compatibility, component contributions, hyper-parameters, and feature-order robustness.
  • Experiments use Criteo, Avazu, and Huawei App Store datasets.
  • Nine baseline models include LR, GBDT, FM, FFM, CCPM, DeepFM, xDeepFM, IPNN, and PIN.
  • Performance is evaluated with AUC and log loss.
  • Experiments are repeated 10 times with different random seeds, followed by two-tailed pairwise t-tests against the best baseline.

3.2 Overall Performance (Q1)

FGCNN achieves the best performance across all three datasets and improves both IPNN and other classifier architectures through automatically generated features.

  • FGCNN achieves the best performance among all evaluated models on all three datasets.
  • 0.05%, 0.14%, and 0.13% AUC improvements over the best baselines occur on Criteo, Avazu, and Huawei App Store, respectively.The corresponding log-loss improvements are 0.09%, 0.24%, and 0.79%.
  • 0.11%, 0.19%, and 0.13% AUC improvements over IPNN occur on Criteo, Avazu, and Huawei App Store, respectively.The corresponding log-loss improvements are 0.2%, 0.29%, and 0.79%.
  • Directly applying CNN in CCPM performs worst among neural models, while FGCNN combines local CNN patterns with global recombination.
  • Generated features improve all tested Deep Classifiers, supporting FGCNN’s compatibility as a general enhancement framework.
  • Removing any FGCNN component reduces performance, and raw plus generated features outperform either feature set alone.

3.5 Hyper-parameter Investigation (Q4)

Hyper-parameter studies show that convolutional depth and generated-feature count generally help, while excessive kernel height or feature generation can hurt; recombination improves robustness to feature-order changes.

  • 3.5.1 Height of Convolutional Kernels: Performance generally rises and then falls as convolutional-kernel height increases.Larger heights capture more features but make sparse interactions harder to learn effectively.
  • 3.5.2 Number of Convolutional Layers: Performance improves as the number of convolutional layers increases, consistent with learning higher-order feature interactions.
  • 3.5.3 Number of Kernels for Generating New Features: Performance gradually improves as more generated features are added, until excess features introduce noise and increase learning difficulty.
  • 3.5.4 Feature-Order Robustness: The Recombination Layer produces better and more stable performance than omitting it when raw-feature order is shuffled.

4 RELATED WORK

CTR research progresses from shallow interaction models to deep models, while FGCNN addresses CNN limitations by generating features before classification.

  • Shallow and deep CTR models: CTR models include shallow approaches such as LR, Poly-2, and FM variants, alongside deep models for learning feature interactions.FM models interactions through latent-vector inner products, while FFM uses multiple latent vectors across feature fields.
  • Deep CTR models: Wide & Deep combines manually engineered wide features with a deep component, whereas DeepFM, IPNN, PIN, and xDeepFM pursue automated interaction modeling.These models differ in whether interactions are represented through FM layers, MLPs, micro-networks, or compressed interaction networks.
  • CNN-based CTR models: CNN-based CTR models explore neighboring feature dependencies, but feature ordering and limited neighbor coverage constrain their learned interactions.CCPM applies convolutions to neighboring fields, while prior work shows that feature arrangement strongly affects CNN performance.
  • FGCNN: FGCNN augments the original feature space with generated features and allows existing state-of-the-art models to serve as its Deep Classifier.Its Recombination Layer combines local CNN patterns to reduce CNN’s weakness in modeling broader interactions.

5 CONCLUSION

FGCNN targets DNN learning difficulty by identifying important features in advance. Experiments report stronger performance across datasets and compatibility with alternative deep classifiers.

  • 5 CONCLUSION: FGCNN combines Feature Generation with a Deep Classifier to identify important features before learning from the augmented feature space.Feature Generation uses CNN to find local patterns and a Recombination Layer to produce global features.
  • 5 CONCLUSION: FGCNN outperforms nine state-of-the-art models on three large-scale datasets.The reported experiments also find better performance when other models are used as Deep Classifiers instead of their original versions.
  • 5 CONCLUSION: Step-by-step experiments show that each FGCNN component contributes to final performance, while the CNN+Recombination structure is more stable than traditional CNNs under feature shuffling.The passage reports that the combined structure performs better and more stably than the traditional CNN structure.
Loading 1904.04447v1…