PaperlayerView exact revision on arXiv

Source-linked AI summary

An Embedding Learning Framework for Numerical Features in CTR Prediction

Huifeng Guo, Bo Chen, Ruiming Tang, Weinan Zhang, Zhenguo Li, Xiuqiang He

arXiv:2012.08986v2cs.IRcs.AIcs.LG

TL;DR

Deep CTR models often concentrate on feature-interaction design, while numerical-feature embeddings face limited capacity or offline, hard discretization. AutoDis learns numerical-feature embeddings through field-specific meta-embeddings, differentiable soft discretization, and aggregation. Online A/B testing reports improvements over the commercial baseline of 2.1% in CTR and 2.7% in eCPM.

  • Problem

    Numerical-feature embedding methods can have low representation capacity or rely on heuristic discretization that is not optimized with the CTR objective.

    Source: §1 · pp. 1–2

  • Method

    AutoDis uses shared field-specific meta-embeddings, differentiable soft discretization, and aggregation to learn numerical-feature representations end to end.

    Source: §1 · p. 2

  • Results

    2.1% CTR and 2.7% eCPM improvements over the commercial baseline were reported in an online A/B test.

    Source: §1 · p. 2

  • Takeaways & Limitations

    AutoDis is a pluggable numerical-feature embedding framework that was deployed on a mainstream advertising platform.

    Source: §1 · p. 2

    Heuristic discretization may separate similar boundary values into different buckets and cannot be optimized jointly with the CTR objective.

    Source: §2.3 · p. 3

Abstract

from arXiv · show

Click-Through Rate (CTR) prediction is critical for industrial recommender systems, where most deep CTR models follow an Embedding \& Feature Interaction paradigm. However, the majority of methods focus on designing network architectures to better capture feature interactions while the feature embedding, especially for numerical features, has been overlooked. Existing approaches for numerical features are difficult to capture informative knowledge because of the low capacity or hard discretization based on the offline expertise feature engineering. In this paper, we propose a novel embedding learning framework for numerical features in CTR prediction (AutoDis) with high model capacity, end-to-end training and unique representation properties preserved. AutoDis consists of three core components: meta-embeddings, automatic discretization and aggregation. Specifically, we propose meta-embeddings for each numerical field to learn global knowledge from the perspective of field with a manageable number of parameters. Then the differentiable automatic discretization performs soft discretization and captures the correlations between the numerical features and meta-embeddings. Finally, distinctive and informative embeddings are learned via an aggregation function. Comprehensive experiments on two public and one industrial datasets are conducted to validate the effectiveness of AutoDis. Moreover, AutoDis has been deployed onto a mainstream advertising platform, where online A/B test demonstrates the improvement over the base model by 2.1% and 2.7% in terms of CTR and eCPM, respectively. In addition, the code of our framework is publicly available in MindSpore(https://gitee.com/mindspore/mindspore/tree/master/model_zoo/research/recommend/autodis).

1 INTRODUCTION

Deep CTR models commonly emphasize feature-interaction architectures, while numerical-feature embeddings remain underexplored despite their importance to prediction. AutoDis addresses limitations of existing numerical-feature representations with end-to-end soft discretization and reports gains in experiments and online testing.

1 INTRODUCTION

  • CTR prediction estimates whether a user will click on a recommended item in a specific context, supporting recommendation decisions.The paper notes its role in recommender systems used for services including web search, commodity recommendation, and online advertising.Source: §1 · p. 1
  • Although most deep CTR research focuses on feature interactions, the embedding module affects their effectiveness and contains many of the model’s parameters.The paper identifies this underexplored module as a key factor in deep CTR models.Source: §1 · p. 1
  • Existing numerical-feature approaches use no embedding, one field embedding, or heuristic discretization, with limited capacity or rules not optimized for CTR prediction.Hard discretization can also give similar values different embeddings or dissimilar values the same embedding.Source: §1 · p. 2
  • AutoDis combines field-specific meta-embeddings, differentiable soft discretization, and aggregation to learn unique numerical-feature representations end to end.Meta-embeddings share global knowledge across values within a field, while soft discretization captures their correlations with those embeddings.Source: §1 · p. 2
  • 2.1% CTR and 2.7% eCPM improvements over the commercial baseline were reported in an online A/B test of AutoDis.Experiments on two public and one industrial dataset also found improvements over existing numerical-feature representation methods, and AutoDis was compatible with popular deep CTR models.Source: §1 · p. 2

2 PRELIMINARY

The section introduces CTR feature embeddings and reviews three ways of representing numerical features. It contrasts their limitations with AutoDis’s stated goal of high-capacity, end-to-end learning with unique representations.

2 PRELIMINARY

  • Training records contain categorical and numerical fields, with categorical embeddings obtained by a lookup operation.The dataset is described as Q labeled instances, each with M categorical fields and N numerical fields.Source: §2 · p. 2

2.1 Category 1: No Embedding

  • No Embedding methods use numerical values or transformations directly, but the authors say their low capacity makes informative field knowledge difficult to capture.Examples include transformed values and a DNN applied to all numerical features.Source: §2.1 · pp. 2–3

2.2 Category 2: Field Embedding

  • Field Embedding multiplies each numerical value by one shared embedding for its field, limiting capacity through sharing and linear scaling.The field vector is uniform across values within that field.Source: §2.2 · p. 3

2.3 Category 3: Discretization

  • Discretization converts numerical values into categories using approaches such as equal-width or equal-frequency buckets, logarithmic mapping, and tree-based methods.The resulting bucket determines an embedding lookup.Source: §2.3 · p. 3
  • The authors identify three discretization limits: separate embeddings for similar boundary values, shared embeddings for dissimilar values, and preprocessing that cannot be optimized with the CTR objective.They say this discretization does not effectively describe continuity in numerical features.Source: §2.3 · p. 3

2.4 Distinction and Relationship

  • AutoDis is presented as a numerical-feature embedding framework designed for high capacity, end-to-end training, and unique representations.The authors contrast it with methods that have low capacity or require offline feature engineering.Source: §2.4 · p. 3

3 METHODOLOGY

AutoDis is a pluggable framework for learning distinctive numerical-feature embeddings end to end, using meta-embeddings, differentiable automatic discretization, and aggregation. Its design aims to balance model capacity, parameter complexity, and continuous-but-different representations.

3.1 Framework Overview

  • AutoDis combines numerical-feature embeddings with categorical embeddings and feeds them to a deep CTR model for prediction.The framework is designed to be compatible with existing deep CTR models.Source: §3 · pp. 3–4, §3.1 · p. 4

3.2 Meta-Embeddings

  • Per-field meta-embeddings share global knowledge across feature values, balancing richer representations against the parameter cost of independent or single field embeddings.The number of meta-embeddings controls model complexity.Source: §3.2 · p. 4

3.3 Automatic Discretization

  • Differentiable soft discretization assigns feature values across buckets according to their correlations with meta-embeddings, avoiding single-bucket hard assignment and enabling end-to-end optimization.The temperature coefficient controls the discretization distribution.Source: §3.3 · pp. 4–5

3.4 Aggregation Function

  • Weighted-Average aggregation uses the bucket correlations to combine meta-embeddings, so relevant embeddings contribute more and each feature receives a unique, continuous-but-different representation.Max-Pooling reintroduces hard selection, while Top-K-Sum does not fundamentally resolve DBS and does not account for correlation values.Source: §3.4 · p. 5

3.5 Training

  • AutoDis is trained jointly with the deep CTR model using LogLoss with an L2 regularization term.The training objective includes parameters for categorical embeddings, AutoDis, and the CTR model.Source: §3.5 · p. 5

4 EXPERIMENTS

The experiments evaluate AutoDis on two public datasets and one industrial dataset, using standard CTR metrics and comparisons with numerical-feature representation baselines. They also test compatibility with six embedding-based deep CTR models.

4.1.1 Dataset and Evaluation Protocols.

  • The evaluation uses Criteo, AutoML, and an industrial dataset, with AUC and LogLoss reported across five random-seed runs.A two-tailed unpaired t-test compares AutoDis with the best baseline.Source: §4.1.1 · p. 5

4.1.2 Baselines and Hyper-parameters Settings.

  • Numerical-feature baselines cover no-embedding, field-embedding, and discretization approaches, while compatibility is tested with six representative deep CTR models.The six models are FNN, Wide & Deep, DeepFM, DCN, IPNN, and xDeepFM.Source: §4.1.2 · p. 6
  • The experiments use mini-batch Adam, tune learning rate and L2 regularization, and set AutoDis to 20 or 40 meta-embeddings on Criteo or AutoML, respectively.The embedding size is 80 on Criteo and 70 on AutoML.Source: §4.1.2 · p. 6

4.2 Overall Performance

AutoDis consistently outperforms numerical-feature representation baselines across three datasets and improves performance when added to popular deep CTR models. Its soft discretization and aggregation address limits of lower-capacity methods and hard discretization.

4.2 Overall Performance

  • AutoDis improves AUC over the best baseline by 0.17%, 0.23%, and 0.22%, respectively, across the three datasets.The comparison uses DeepFM and reports results in Table 3.Source: §4.2.1 · p. 6, Table 3 · p. 6

4.2.1 Compared with Other Representation Learning Methods.

  • No Embedding and Field Embedding have lower performance than Discretization and AutoDis, while heuristic discretization cannot be optimized with CTR models.AutoDis instead combines shared meta-embeddings with differentiable soft discretization and aggregation to learn Continuous-But-Different representations.Source: §4.2.1 · p. 6
  • AutoDis significantly improves prediction performance over Field Embedding when applied to a series of popular deep CTR models.The numerical-feature discretization and embedding learning are optimized with the CTR models' ultimate prediction objective.Source: §4.2.2 · p. 6, Table 4 · p. 6

4.3 Deployment & Online A/B Testing

In an advertising-platform test, AutoDis improved offline AUC and online CTR and eCPM relative to the control group. Embedding analyses show distinct representations for individual values and similar embeddings or discretization distributions for nearby values.

4.3 Deployment & Online A/B Testing

  • The online test used a mainstream advertising platform with millions of daily active users; its control group used manually designed discretization rules.The experience group used AutoDis for numerical-feature discretization and embedding learning.Source: §4.3 · pp. 6–7
  • AutoDis improved offline AUC by 0.2% and online CTR and eCPM by 2.1% and 2.7%, respectively, versus the control group.The online gains were reported as statistically significant.Source: §4.3 · p. 7

4.4 Embedding Analysis

  • AutoDis gives each numerical feature a unique embedding, with similar-valued features close together, unlike EDD's identical within-bucket embeddings.The visualization uses 250 embeddings from Criteo's third numerical field, projected with t-SNE.Source: §4.4.1 · p. 7, Figure 5 · p. 7

4.4.1 Macro-Analysis of Embeddings.

  • Adjacent values 1 and 2 have similar softmax discretization distributions, while distant values 1 and 10 have diverse distributions.The authors report that this pattern is conducive to preserving embedding continuity.Source: §4.4.2 · p. 7, Figure 6 · p. 7

4.5 Numerical Fields Analysis

Adding numerical fields one at a time to the categorical-field baseline shows that AutoDis improves performance with a single field and gains cumulatively with multiple fields.

4.5 Numerical Fields Analysis

  • AutoDis improves prediction even when only one numerical field is added to the 26 categorical fields.The Criteo analysis added numerical fields cumulatively, in the dataset's original order and a random order.Source: §4.5 · p. 7
  • Adding multiple numerical fields produces cumulative performance improvement with AutoDis.The analysis evaluates fields in both the original and a random order.Source: §4.5 · p. 7
  • Compared with existing methods, AutoDis's performance improvement is more significant and stable.Source: §4.5 · p. 7

4.6 Model Complexity

Compared with EDD, AutoDis adds negligible model parameters and has acceptable inference time. Its end-to-end framework avoids the extra efficiency overhead associated with manual discretization design or training another model.

4.6 Model Complexity

  • The increase in model parameters with AutoDis is negligible compared with EDD.The comparison is reported on Criteo using DeepFM.Source: §4.6 · p. 7, Table 5 · p. 8
  • AutoDis achieves acceptable inference time compared with EDD, with model parameters and batch inference time compared in Table 5.Source: §4.6 · pp. 7–8, Table 5 · p. 8
  • AutoDis's end-to-end framework adds no extra efficiency overhead, unlike EDD's manual design and preprocessing or TD's additional GBDT training.Source: §4.6 · p. 8

4.7 Ablation Study and Hyper-Parameters Sensitivity

The ablations compare aggregation strategies, temperature settings, and meta-embedding counts. Weighted-Average performs best among the tested aggregation strategies, while adaptive temperature and a moderate number of meta-embeddings are favored in the reported comparisons.

4.7.1 Aggregation Strategy.

  • Weighted-Average achieves the best prediction performance among the tested aggregation strategies.The authors attribute this to using meta-embeddings and their correlations, overcoming the DBS and SBD problems compared with the other strategies.Source: §4.7.1 · p. 8

4.7.2 Temperature Coefficient.

  • The adaptive temperature network achieves the best results compared with a global temperature coefficient.The reported optimal global temperatures are around 1e-5 on Criteo and 4e-3 on AutoML; Figure 9 compares the settings.Source: §4.7.2 · p. 8, Figure 9 · p. 8

4.7.3 Number of Meta-Embeddings.

  • Increasing meta-embeddings initially improves performance, but too many increase computational burden and frustrate performance.Considering prediction effect and training efficiency, the authors use 20 on Criteo and 40 on AutoML; Figure 10 shows the performance comparison.Source: §4.7.3 · p. 8, Figure 10 · p. 8

5 RELATED WORK

Related work covers both embedding and feature-interaction components of deep CTR models. It notes that numerical-feature embedding receives little attention, while interaction research develops stacked and parallel architectures.

5 RELATED WORK

  • Most deep CTR models use an Embedding & Feature Interaction paradigm, with embeddings feeding interaction modules that model feature relationships.The interaction module may model implicit or explicit, low- or high-order interactions.Source: §5 · p. 8

5.1 Embedding

  • Embedding research explores adaptable embedding algorithms and compression, but these methods rely on look-up tables for categorical features or discretized numerical features.The passage identifies few studies of embedding learning for numerical features.Source: §5.1 · p. 8

5.2 Feature Interaction

  • Feature-interaction models use stacked or parallel structures, pairing DNN-based implicit interactions with varied explicit-interaction mechanisms.Examples include FM structures in DeepFM and AutoFIS, DCN's cross network, xDeepFM's CIN, and AutoInt's multi-head self-attention.Source: §5.2 · p. 9

6 CONCLUSION

AutoDis is a pluggable framework for learning numerical-feature embeddings, using meta-embeddings, differentiable discretization, and weighted-average aggregation. Experiments on three real-world datasets and an online A/B test are reported as demonstrating its effectiveness and compatibility.

6 CONCLUSION

  • AutoDis shares meta-embeddings across intra-field features to capture global informative knowledge.The paper describes this as a source of high model capacity.Source: §6 · p. 9
  • Differentiable automatic discretization performs soft discretization, allowing AutoDis to be optimized jointly with the deep CTR model's ultimate goal.Source: §6 · p. 9
  • Weighted-average aggregation gives each feature a unique Continuous-But-Different representation.The paper reports extensive experiments on three real-world datasets and an online A/B test as demonstrating effectiveness and compatibility.Source: §6 · p. 9

Related papers

Loading 2012.08986v2…