PaperlayerView exact revision on arXiv

Source-linked AI summary

Open Benchmarking for Click-Through Rate Prediction

Jieming Zhu, Jinyang Liu, Shuai Yang, Qi Zhang, Xiuqiang He

arXiv:2009.05794v6cs.IRcs.AI

TL;DR

CTR research lacks standardized evaluation, making published results difficult to reproduce and compare. This paper builds BARS-CTR and rigorously re-evaluates existing models under uniform protocols. The results show that many models differ less than expected after tuning, while real modeling progress remains challenging.

  • Problem

    CTR studies lack standardized benchmarks and uniform protocols, limiting reproducibility and comparability despite CTR’s importance in industry.

  • Method

    The paper builds BARS-CTR, standardizes evaluation protocols, and re-evaluates 24 models across two datasets and multiple settings.

  • Results

    Many models differ less than expected after extensive tuning, and existing reported results can be inconsistent.

  • Takeaways & Limitations

    The released code, protocols, hyper-parameters, and results support reproducible comparisons and help researchers assess new CTR models.

  • Takeaways & Limitations

    The benchmark covers only Criteo and Avazu, whose anonymized data lack explicit user and item fields needed by some models.

Abstract

from arXiv · show

Click-through rate (CTR) prediction is a critical task for many applications, as its accuracy has a direct impact on user experience and platform revenue. In recent years, CTR prediction has been widely studied in both academia and industry, resulting in a wide variety of CTR prediction models. Unfortunately, there is still a lack of standardized benchmarks and uniform evaluation protocols for CTR prediction research. This leads to non-reproducible or even inconsistent experimental results among existing studies, which largely limits the practical value and potential impact of their research. In this work, we build an open benchmark for CTR prediction, namely BARS-CTR, and present a rigorous comparison of different models in a reproducible manner. To this end, we ran over 7,000 experiments for more than 12,000 GPU hours in total to re-evaluate 24 existing models on multiple datasets and settings. Surprisingly, our experiments show that with sufficient hyper-parameter search and model tuning, many deep models have smaller differences than expected. The results also reveal that making real progress on the modeling of CTR prediction is indeed a very challenging research task. We believe that our benchmarking work could not only allow researchers to gauge the effectiveness of new models conveniently but also make them fairly compare with the state of the arts. We have publicly released the benchmarking code, evaluation protocols, and hyper-parameter settings of our work to promote reproducible research in this field.

1 INTRODUCTION

CTR prediction is important but difficult because industrial data are large, sparse, and heterogeneous. The paper addresses inconsistent evaluation by proposing an open, reproducible benchmark for comparing CTR models.

  • Motivation: CTR prediction affects applications including recommendation, advertising, and product search, where small accuracy gains can have substantial practical value.An absolute improvement of 1‰ in logloss or AUC is considered practically significant in reported industrial studies.
  • Existing Models: CTR research spans models from logistic regression, factorization machines, and decision trees to numerous deep neural networks.Examples include Wide&Deep, DeepFM, DCN, xDeepFM, FiBiNET, and DIN.
  • Reproducibility Gap: Missing standardized benchmarks and evaluation protocols make results difficult to reproduce and comparisons between papers unreliable.Differences in data splits, preprocessing, random seeds, and undocumented baseline implementations contribute to these problems.
  • Contribution: The paper proposes BARS-CTR, an open benchmarking pipeline that rigorously re-evaluates 24 existing models under a uniform setup.The study reports over 7,000 experiments and more than 12,000 GPU hours across multiple settings of the Criteo and Avazu datasets.
  • Contribution: The benchmark provides code, evaluation protocols, hyper-parameter settings, and results to support reproducible and fair CTR research.These resources are intended to help researchers assess new models and analyze existing model strengths and bottlenecks.

2 CTR PREDICTION

CTR prediction estimates click probabilities from large, sparse, multi-field tabular data. The section describes feature representations, interaction modeling, the binary cross-entropy objective, and representative shallow and deep architectures.

  • Data and Features: CTR prediction uses user, item, and context features that may be numerical, categorical, or multi-valued.Sparse features commonly produce high-dimensional spaces after one-hot or multi-hot encoding.
  • Feature Representation: Feature embedding maps sparse categorical features into low-dimensional dense vectors, while numeric features can be bucketized or otherwise embedded.For a categorical field, the paper gives e_i = V_i x_i, with embedding dimension d and vocabulary size n.
  • Feature Interactions: Feature interactions are central to improving CTR classification, with factorization machines using inner products to capture pairwise interactions.The broader model literature explores increasingly complex interaction structures.
  • Loss Function: Binary cross-entropy, or logloss, evaluates predicted click probabilities against ground-truth labels, with lower logloss indicating better performance.The model produces an estimated click probability through a sigmoid applied to its learned function of the input features.
  • Representative Models: Shallow models remain useful for large-scale industrial CTR tasks because of their simplicity and efficiency, while deep models target sophisticated high-order nonlinear interactions.The surveyed families include LR, FM, FFM, HOFM, DNN, Wide&Deep, DeepFM, DCN, xDeepFM, and attention-based models.

3 OPEN CTR BENCHMARKING

The section introduces the paper’s reproducibility requirements, standardized evaluation protocols, open-source benchmarking toolkit, and analysis of benchmarking results.

  • Method: The benchmarking methodology begins by defining reproducibility requirements for CTR research.The section frames these requirements as a basis for evaluating existing studies and their available artifacts.
  • Method: It then specifies evaluation protocols and provides an open-source toolkit for benchmarking CTR prediction models.These components support a uniform and reusable benchmarking pipeline.
  • Results Analysis: Finally, the paper reports and analyzes the resulting benchmark comparisons.The analysis connects reproducibility requirements and evaluation procedures with empirical model results.

3.1 Reproducibility Requirements

Reproducible CTR research requires standardized procedures and accessible artifacts across data, models, and evaluation. The paper argues that a standardized benchmarking pipeline is needed to enable fair comparison.

  • Reproducibility Requirements: Reproducibility requires checking whether each evaluation step and its artifacts are totally, partially, or not available.The paper marks requirements with ✓, −, and × to indicate complete, partial, or absent compliance.
  • Data preprocessing: Missing data-splitting scripts, random seeds, and preprocessing details can make results across studies incomparable.Unknown handling of numeric features and rare categorical features further complicates replication.
  • Model source code: Model source code often omits training procedures or key dataset-specific hyper-parameters needed for reproducibility.Third-party implementations may be available without complete training workflows.
  • Baseline hyper-parameters: Baseline hyper-parameters should be exhaustively tuned to support fair model comparisons.Existing studies often report inconsistent baseline results because preprocessing and implementations are unknown.
  • Benchmarking pipeline: The proposed benchmarking pipeline aims to standardize evaluation and provide comprehensive open results for CTR prediction.Its stated purpose is to enable reproducible research and fairness of comparison.

3.2 Evaluation Protocol

The evaluation protocol uses widely adopted real-world datasets, fixed data splits and preprocessing, common metrics, and an open workflow designed for reproducibility and tuning. The framework records settings and supports systematic hyper-parameter search.

  • Datasets: Criteo and Avazu are used as the main evaluation datasets because they are open production click-log datasets with tens of millions of samples.The benchmark also reports results on additional datasets through the BARS website.
  • Data splits: Both datasets are randomly split 8:1:1 using AutoInt code and random seed 2018 for exactly reproducible comparisons.The resulting splits are named Criteo_x4 and Avazu_x4.
  • Preprocessing: The benchmark follows AutoInt preprocessing while modifying it and fixing a defect to improve benchmark results.The protocol includes dataset-specific preprocessing changes for Criteo and Avazu.
  • Metrics: AUC and logloss are the benchmark metrics, with higher AUC and lower logloss indicating better CTR prediction.AUC measures positive-versus-negative ranking probability, while logloss is logistic loss or binary cross-entropy.
  • Reproducible workflow: The complete workflow covers preprocessing, loading, training, early stopping, learning-rate decay, hyper-parameter search, seeding, and logging.FuxiCTR integrates these components so researchers can reuse code, build models, and add datasets.
  • Reproducibility controls: The protocol records split checksums, fixes experiment seeds, stores settings in configuration files, and uses PyTorch to reduce GPU non-determinism.All evaluation settings and results are open sourced.

3.3 Results Analysis

Benchmarking exposes inconsistencies in previously reported results and shows that extensive retuning materially affects performance. Under a common protocol, differences among many models are smaller than expected.

  • Benchmark setup: Approximately 73 runs per model on average indicate that the benchmark models were extensively tuned.Results cover logloss, AUC, training time, epochs, and parameter counts across four dataset settings.
  • Reported-result inconsistency: Previously reported results are inconsistent, with InterHAt underperforming LR on both datasets and DeepCross underperforming LR on Avazu.The paper attributes these discrepancies largely to different data splits and preprocessing procedures.
  • Retuning results: After retuning, the benchmark generally obtains better performance than the best reported results.The authors connect large improvements under shared splitting to modified preprocessing that produces stronger baselines.
  • Model differences: Exhaustive retuning finds smaller-than-expected differences among many deep CTR models.The comparison is conducted under the same evaluation protocol.
  • Hyper-parameter retuning: Up to 5‰ improvement is achieved after retuning over original hyper-parameters on new data settings.The paper notes that a 1‰ improvement is considered practically significant for industrial CTR prediction.
  • Fair comparison: A common benchmark is needed because following original hyper-parameters while changing data splits undermines fair comparison.The issue remains even when studies intend to preserve fairness by reusing baseline settings.

3.4 Key Factors to Performance Tuning

Performance tuning depends strongly on preprocessing choices and batch size. The paper highlights category-frequency filtering and large batches as practical tuning factors.

  • Data preprocessing: Data preprocessing can determine a model's performance upper bound.The paper states that existing work seldom tunes min_counts thresholds for categorical features.
  • Data preprocessing: Choosing an appropriate threshold for filtering infrequent categorical features yields much better performance.The paper identifies min_counts tuning as an overlooked preprocessing factor.
  • Batch size: A large batch size usually makes training faster and improves model performance on large CTR datasets.

4 DISCUSSION

The benchmark’s scope is limited by its datasets, random data splits, training-time efficiency measure, and grid-search hyper-parameter tuning. The authors identify broader datasets, temporal evaluation, inference-time benchmarking, and advanced auto-tuning as future directions.

  • More datasets: The benchmark uses only Criteo and Avazu, whose anonymization omits explicit user and item fields needed by some models.Consequently, models requiring explicit user-item interactions or user behavior sequences cannot be benchmarked on these datasets.
  • Data splits: The benchmark randomly splits data because train, validation, and test distributions are more consistent under this setting.The authors plan to evaluate sequential time-based splits and apply CTR calibration when necessary.
  • Efficiency benchmarking: Efficiency is mainly evaluated through training time, while inference time remains a desirable future benchmark for real-time CTR applications.Inference latency matters under strict real-time constraints when practitioners choose models.
  • Auto-tuning of hyper-parameters: The benchmark mainly uses grid search, although quickly finding optimal hyper-parameters and retuning them for evolving data remains open.The authors suggest advanced AutoML methods such as Bayesian optimization for future tuning improvements.

5 RELATED WORK

CTR research spans linear, factorization-machine, tree-based, deep, sequence, multi-task, and multi-modal approaches. The paper situates its benchmark within the need for standardized protocols and reproducible comparisons as new models proliferate.

  • Feature interaction learning: CTR models evolved from linear methods and factorization machines to deep learning, with newer architectures targeting diverse interaction patterns and network components.Examples include explicit interaction, convolutional, recurrent, and attention-based models.
  • Feature interaction learning: Feature-interaction research addresses limitations of linear models through feature conjunctions, pairwise interactions, field awareness, interaction importance, robustness, and interpretability.Factorization-machine extensions still face difficulty capturing high-order feature interactions.
  • Behaviour sequence modeling: Behavior-sequence models use attention, LSTM, GRU, and memory networks to capture users’ historical behavior when predicting clicks.Representative models include DIN, DIEN, DSIN, HPMN, and DSTN.
  • Multi-task learning: Multi-task models exploit browsing, favorites, add-to-cart, and purchase feedback to learn relationships among user behaviors for CTR prediction.Examples include ESMM, MMoE, and PLE.
  • Multi-modal learning: Multi-modal CTR research incorporates item text, images, and video, but using such information remains an important problem requiring further exploration.Early studies report benefits from incorporating multi-modal content features.
  • Benchmarking and Reproducibility: The paper addresses rapid model proliferation and inconsistent evaluations by releasing standardized protocols, detailed configurations, running logs, and benchmark results for over 20 models.These artifacts are intended to support reproducibility and fair comparison with baselines.

6 CONCLUSION

The paper presents an open CTR benchmark designed to reduce non-reproducible and inconsistent results. Across extensive evaluations, many models differed less than expected, while the benchmark provides resources for reproducible research and learning.

  • Conclusion: The benchmark standardizes evaluation protocols and compares 24 existing models across two real-world datasets using over 7,000 experiments and more than 12,000 GPU hours.The authors describe it as the first open benchmark for CTR prediction.
  • Conclusion: Many CTR models differ less than expected, and inconsistent results exist across prior papers.The benchmark is intended to support more reproducible research and help beginners learn state-of-the-art CTR models.
Loading 2009.05794v6…