PaperlayerView exact revision on arXiv

Source-linked AI summary

Click-Through Rate Prediction in Online Advertising: A Literature Review

Yanwu Yang, Panyu Zhai

arXiv:2202.10462v1cs.IRcs.AIcs.LG

TL;DR

CTR prediction research has expanded rapidly, yet its methodological evolution and framework relationships remain insufficiently synthesized. This paper conducts a systematic literature review focused on modeling frameworks, classifies and compares advertising CTR models, and identifies trends, challenges, and future directions.

  • Problem

    Existing CTR prediction studies address specific algorithmic problems but provide few comprehensive syntheses of methodological evolution and connections among modeling frameworks.

  • Method

    The paper systematically reviews advertising CTR prediction research, classifying frameworks, variants, ensembles, evaluations, feature interactions, and benchmark comparisons.

  • Results

    The review organizes approximately 175 CTR prediction models developed or experimentally compared over 9 public datasets and synthesizes their performance evaluation.

  • Takeaways & Limitations

    The review provides fundamental knowledge and an entry point for scholars studying advertising CTR prediction.

  • Takeaways & Limitations

    Comparison studies face challenges from computational protocols involving data preprocessing, sampling, feature engineering, and experimental setups.

Abstract

from arXiv · show

Predicting the probability that a user will click on a specific advertisement has been a prevalent issue in online advertising, attracting much research attention in the past decades. As a hot research frontier driven by industrial needs, recent years have witnessed more and more novel learning models employed to improve advertising CTR prediction. Although extant research provides necessary details on algorithmic design for addressing a variety of specific problems in advertising CTR prediction, the methodological evolution and connections between modeling frameworks are precluded. However, to the best of our knowledge, there are few comprehensive surveys on this topic. We make a systematic literature review on state-of-the-art and latest CTR prediction research, with a special focus on modeling frameworks. Specifically, we give a classification of state-of-the-art CTR prediction models in the extant literature, within which basic modeling frameworks and their extensions, advantages and disadvantages, and performance assessment for CTR prediction are presented. Moreover, we summarize CTR prediction models with respect to the complexity and the order of feature interactions, and performance comparisons on various datasets. Furthermore, we identify current research trends, main challenges and potential future directions worthy of further explorations. This review is expected to provide fundamental knowledge and efficient entry points for IS and marketing scholars who want to engage in this area.

1. Introduction

Online advertising CTR prediction has attracted sustained research attention, but existing work offers limited synthesis of methodological evolution and connections among modeling frameworks. This review systematically organizes advertising CTR prediction models, evaluates their performance, and identifies research trends, challenges, and future directions.

  • CTR prediction is widely studied because click-based measures reflect advertisement relevance from users’ perspectives and improving CTR supports sustainable online advertising development.
  • Existing studies provide algorithmic detail for specific CTR problems, but the evolution and connections among modeling frameworks remain insufficiently synthesized.
  • The review systematically surveys CTR prediction research with special attention to modeling frameworks.
  • It identifies current research trends, major challenges, and potential future directions, while focusing on online advertising rather than recommender systems or Web search.
  • It classifies model categories by presenting basic frameworks, variants, ensemble models, advantages, disadvantages, and performance evaluations across datasets.
  • The review summarizes models by feature-interaction complexity and order, while covering state-of-the-art and latest advertising CTR prediction models.

2. Literature Search and Study Identification

The review searched six major academic databases using advertising CTR-related keywords, supplemented results through citation searches and manual screening. The process produced 132 included articles and showed sharply increasing publication activity, especially during 2016–2020.

  • The search covered Web of Science, ACM, IEEE, EBSCOhost, ScienceDirect, and ABI/Inform Global using full-text keyword searches.
  • Initial database searches returned 101 Web of Science, 246 ACM, 106 IEEE, 138 EBSCOhost, 30 ScienceDirect, and 47 ABI/Inform Global results.
  • After removing duplicates and non-article materials, 468 articles remained for broader screening.
  • Citation retrieval in Google Scholar added 20 results, while manual title, abstract, full-text, and dataset screening excluded 356 articles.
  • The final review included 132 articles: 26 peer-reviewed journal articles, 93 conference articles, and 13 pre-prints.
  • Publications increased exponentially since 2007 and peaked during 2016–2020, indicating substantial research activity in CTR prediction.

3. Preliminaries

The preliminaries define CTR prediction, describe feature categories and engineering approaches, and review commonly used evaluation metrics.

  • 3.1 The Definition of CTR Prediction Problem: CTR prediction estimates whether a user will click a particular advertisement from user–advertisement instances with observed binary click labels.The predicted probability is denoted as CTR and is modeled as p = f(u, x).
  • 3.1 The Definition of CTR Prediction Problem: Models commonly learn CTR by minimizing cross-entropy between binary target labels and predicted probabilities.The predicted CTR p_i lies in [0,1].
  • 3.2 Features for CTR Prediction: CTR prediction features are categorized into advertising, user, context, query, and publisher features.These categories summarize the feature types used in the literature.
  • 3.2 Features for CTR Prediction: Feature engineering transforms raw data into useful predictors manually or automatically, with counting features addressing cold-start and data-drift problems.Traditional manual engineering requires substantial time and expert experience, whereas models such as PNN and DeepFM enable automatic feature engineering.
  • 3.3 Evaluation Metrics for CTR Prediction: CTR prediction studies use metrics including precision, recall, F1-score, accuracy, AUC-ROC, AUC-PR, RelaImpr, Logloss, MSE, RMSE, RIG, and field-level calibration errors.Table 3 provides metric definitions and related studies.

4. State-of-the-art CTR Prediction Models

The review categorizes state-of-the-art CTR prediction models into multivariate statistical, factorization-machine, deep-learning, and tree-model groups, comparing their frameworks, trade-offs, and performance. Across these groups, model development increasingly targets feature interactions, including higher-order interactions, attention, field awareness, and sequential behavior.

  • Model categories: State-of-the-art CTR prediction models are categorized into multivariate statistical models, factorization-machine models, deep-learning models, and tree models.The review examines each category's modeling frameworks, advantages, disadvantages, and CTR-prediction performance.
  • Multivariate statistical models: LR is widely used and outputs click probabilities, but its independence assumption prevents it from representing interactive effects and nonlinear feature-label relationships.LR commonly serves as a baseline and can also form the top layer after feature selection and feature interactions.
  • Feature interactions: Manual feature interactions depend on experts' domain knowledge because no commonly accepted procedure exists, motivating models that derive feature relationships automatically.Feature interactions can generate useful features and significantly improve CTR-prediction performance.
  • Factorization-machine models: Poly2 explicitly models pairwise feature interactions, while factorization machines estimate them through parameter factorization and improve RMSE relative to Poly2.Poly2 combines individual and combinatorial features; FMs are described as estimating interactions reliably for advertising CTR prediction.
  • Factorization-machine models: Field-aware, neural, and attentive factorization models extend interaction modeling, with reported gains across AUC, Logloss, and RMSE against earlier factorization-based baselines.FFMs and FwFMs outperform several simpler baselines, while HoAFM outperforms FMs and NFM on benchmark advertising datasets.
  • Deep-learning models: Deep CTR models combine low- and high-order interactions, and reported comparisons favor DeepFM, xDeepFM, and related models over several linear, factorization, and deep-learning baselines.DeepFM explicitly models low-order interactions and implicitly models high-order interactions in one framework; xDeepFM outperforms DeepFM on CTR-prediction datasets.

4.4 Tree models

Tree-based CTR models use boosting to build predictive ensembles, with GBDT and XGBoost as representative frameworks. Their extensions improve feature selection or scalability, but tree models face efficiency and high-dimensional sparsity limitations.

  • GBDT: GBDT builds a tree ensemble by sequentially adding base learners that reduce the loss.Its base learner can be a regression tree, and each step follows a greedy descent direction.
  • GBDT: GBDT handles heterogeneous features, collinearity, nonsparse data, missing data, and automatic feature selection while retaining interpretability.The model explicitly characterizes contributions from different features.
  • Limitations and extensions: GBDT ensembles consume substantial memory, train slowly because learning is sequential, and perform ineffectively on sparse categorical data.All base learners must be evaluated, and tree-based processing can produce highly sparse feature vectors.
  • Extensions: Ensembles such as GBDT+LR, GBDT+gcForest, GBDT+DNN, and XGBDeepFM combine tree-based feature selection with other prediction components.XGBDeepFM additionally uses factorization machines for second-order interactions and a DNN for high-order representations.
  • XGBoost: XGBoost extends GBDT by using first- and second-order gradients and regularization to optimize an additive tree model.Its greedy split procedure evaluates candidate loss reductions, and parallel splitting can accelerate exploration.
  • XGBoost: XGBoost matched scikit-learn in AUC with shorter per-tree training time, while both XGBoost and scikit-learn outperformed R.gbm in AUC.These comparisons were reported by Chen and Guestrin (2016).
  • Extensions: LightGBM improved substantially over GBDT and XGBoost in computing speed and memory consumption for high-dimensional, large datasets.It uses gradient-based one-side sampling, exclusive feature bundling, and leaf-wise growth with depth control.

5. Discussions

The review organizes CTR models by complexity and feature-interaction order, then compares them across datasets and metrics. Evidence generally favors higher-order and extended models, but several comparisons remain inconsistent, especially between DeepFM and PNN.

  • Model classification: CTR models are grouped into low-order models with interactions up to order 2 and high-order models with interactions above 2.The review compares these groups by complexity, interaction order, advantages, disadvantages, and performance.
  • Low-order models: FMs-based models target sparse data: FMs model second-order interactions, FFMs distinguish field pairs, and FwFMs reduce parameters and computational complexity.The review states that FwFMs retain prediction performance while reducing complexity.
  • Performance comparisons: Among 62 studies across 7 public and 2 proprietary datasets, most reported that FMs outperformed LR and Poly2 across AUC, Logloss, RMSE, accuracy, RelaImpr, and RIG, with exceptions.The exceptions included studies where FMs underperformed LR or Poly2.
  • Performance comparisons: Among 28 studies, most reported that FFMs outperformed FMs, while 6 of 8 studies found FFMs outperformed FwFMs in AUC and Logloss.Two studies found FwFMs better than FFMs when using the same number of parameters.
  • High-order models: Deep models generally outperformed FMs, with FNN and PNN usually achieving better AUC, Logloss, RMSE, and related metrics.The review reports exceptions for both FNN-versus-FM and PNN-versus-FM comparisons.
  • High-order models: DeepFM and xDeepFM generally outperformed several deep baselines, including FNN, Wide & Deep, and CCPM.The improvement of xDeepFM over DeepFM was largely confirmed across 23 studies, although exceptions were reported.
  • Contested comparisons: PNN versus DeepFM results were controversial: 16 of 28 studies favored PNN, 6 favored DeepFM, and 6 were inconsistent.Among studies specifying IPNN, 11 favored IPNN and 2 supported DeepFM; many studies did not identify the PNN variant.
  • Overall findings: Overall, high-order models usually performed better than low-order models, but inconsistent and opposing evidence remained in some comparisons.The review identifies DeepFM-versus-PNN comparisons as especially controversial.

6. Research Perspectives

The review identifies research directions around richer feature interactions, numerical-feature representation, scalable training, interpretability, and persistent data challenges. It also highlights graph-based methods and multimodal interactions as potential avenues for future CTR prediction research.

  • Feature interactions: Feature interactions remain central because modeling correlations among features can enhance CTR prediction.Matrix weighting and attention mechanisms are proposed for explicit and interpretable interactions, including higher-order extensions of FMs.
  • Feature representation: Numerical features are comparatively overlooked, motivating embeddings with meta-embeddings, differentiable discretization, and aggregation for continuous representations.Another approach transforms numerical features into categorical values using tree models before learning embeddings.
  • Ensemble modeling: Ensemble research combines GBDT-based feature selection with gcForest, DNNs, or DeepFM, while other models combine low- and high-order components.FM-DNN ensembles are described as an active research direction.
  • Scalable training: Embedding-based deep CTR models create major memory and communication demands because sparse inputs can require hundreds of GB or even TB.Host-GPU synchronization latency is identified as a serious concern for media providers.
  • Scalable training: MixCache-based systems and dynamic communication thresholding address distributed-training costs through embedding synchronization and gradient or activation compression.These methods target communication costs in large-scale hybrid or data-parallel training.
  • Interpretability: Existing models face interpretability and feature-engineering limitations: manual engineering harms replicability, while implicit high-order interactions are difficult to explain.The review also notes that not all combinatorial features improve performance and that explicit high-order models need stronger theoretical support.
  • Data challenges: CTR prediction must address label imbalance, heterogeneous samples, cold-start advertisements, and sparse categorical inputs.These issues can make minority-click classification difficult, limit embedding quality for new advertisements, and increase computational requirements.
  • Future directions: Graph neural networks could combine richer feature interactions and connect advertisements to transfer information for cold-start prediction.The review specifically proposes integrating FwFM, FmFM, or AOAFM within graph structures and aggregating information across advertisements.

7. Conclusion

The conclusion presents advertising CTR prediction as an ongoing research area and summarizes the review’s coverage, evidence base, challenges, and future directions.

  • Advertising CTR prediction is expected to remain a major research topic in the coming decades.
  • The review organizes state-of-the-art CTR prediction models around modeling frameworks, their advantages and disadvantages, and performance evaluation.
  • The review identifies research trends, main challenges, future directions, and benchmark comparisons for evaluating CTR prediction models.
  • Comparison studies face challenges from inconsistent results and differences in data preprocessing, sampling, feature engineering, and experimental setups.
  • About 175 CTR prediction models were developed or experimentally compared across 9 public datasets and dozens of proprietary datasets.
  • The review is intended to provide fundamental knowledge and entry points for IS and marketing scholars and a theoretical basis for developing novel CTR models.

Type

The supplied passage identifies Table 3 as presenting evaluation metrics for CTR prediction.

  • Table 3 presents evaluation metrics for CTR prediction.

Category Modeling Approach

The review organizes CTR prediction research by modeling framework, feature-interaction order, complexity, limitations, and empirical performance across datasets. It covers statistical, factorization-machine, neural, tree-based, and hybrid approaches, with comparisons showing varied rankings across metrics and datasets.

  • Model categories: CTR prediction models are classified into statistical, factorization-machine, neural, tree-based, and related framework categories.The review also compares models by parameter count and order of feature interactions.
  • Feature interactions: Models differ in feature-interaction order, with low-order models capturing interactions up to order two and high-order models exceeding order two.The review uses this distinction to organize model complexity and representational capacity.
  • Model limitations: Linear models can miss interactive effects and nonlinear relationships, while Poly2 is slower and performs less well on sparse data.Additional limitations include FwFMs lacking high-order interactions, LSTM requiring more training time and memory, and FNN constraints from pretrained FM representations.
  • Performance comparisons: Reported comparisons span AUC, Logloss, RMSE, MSE, accuracy, precision, recall, F1-measure, RIG, RI, NE, and relative-improvement measures.The review summarizes results from multiple advertising datasets, including Taobao, Huawei DIGX, proprietary data, and Criteo-related evaluations.
  • Performance comparisons: Across datasets, advanced models frequently rank above simpler baselines, but the ordering varies by metric, dataset, and compared model family.Examples include DeepFM, xDeepFM, PNN, AutoInt, graph-based models, and other extensions outperforming or ranking above LR and FM in reported AUC or Logloss orderings.
  • Model extensions: The literature includes hybrid and automated frameworks that combine deep, tree, interaction, attention, graph, or feature-engineering components.Examples include Deep&Cross, xDeepFM, AutoCTR, DeepLight, GraphFM, and XGBDeepFM.

Appendix for “Click-Through Rate Prediction in Online Advertising: A Literature Review”

The appendix identifies the authors’ institutional affiliation and provides their contact email addresses.

  • Author information: The authors are affiliated with the School of Management at Huazhong University of Science and Technology in Wuhan, China.The appendix lists the institutional affiliation and corresponding email addresses.

A.1 Web Search, Recommender System and Online Advertising

Web search, recommender systems, and online advertising all match users to information objects, but they differ in inputs, outputs, and goals. Advertising uses user profiles, behaviors, contexts, and advertisement characteristics to learn clicking behavior and maximize campaign outcomes.

  • Shared architecture: Despite their differences, search, recommendation, and advertising share similar data-processing procedures and system architectures.Their commonality is matching explicit or implicit user needs to collections of information objects.
  • Inputs: Search uses explicit queries, whereas recommender systems and advertising typically infer users’ implicit interests from behaviors and contexts.All three mechanisms match user needs to information objects such as Web pages, products, or advertisements.
  • Outputs: Search returns relevant results, recommender systems suggest products, and advertising presents sponsored information matched to users.Advertising and recommendation both process commercial information, while search primarily provides organic results.
  • Goals: Search seeks relevant and diverse retrieval, recommendation seeks personalized product suggestions, and advertising seeks higher CTR, conversion rate, revenue, or ROI.The three mechanisms therefore share matching processes but pursue different operational goals.
  • Advertising CTR prediction: CTR prediction in advertising learns users’ clicking behavior toward advertisements from advertising-object and contextual features.Advertisement position, quality, slogan, and related characteristics may influence click probability.

A.2 The Procedure of CTR Prediction

The CTR prediction procedure preprocesses data, engineers features, builds a model, trains and validates it with train-test data, and then predicts click probabilities for new samples.

  • Data preprocessing: The procedure begins by cleaning abnormal data, filling missing values, and removing duplicate samples.These preprocessing steps prepare the raw dataset for subsequent feature engineering and modeling.
  • Feature engineering: Feature engineering selects and extracts useful prediction features from raw-data attributes.This is the second step of the CTR prediction procedure.
  • Model construction: The CTR model is built with statistical or machine-learning techniques.The review elaborates the model families and frameworks in its main modeling section.
  • Training and validation: The dataset is divided into training and test subsets, model parameters are optimized on training data, and performance is evaluated on test data.Stochastic Gradient Descent is given as an example optimization algorithm.
  • Prediction: The trained CTR prediction model computes click probabilities for a new set of samples.This is the final output of the five-step procedure.

A.3 Research Articles included in This Review

This section lists the research articles included in the review, spanning work on search, display, social-media, mobile, and contextual advertising.

  • Research coverage: The review’s included-article list covers studies published across multiple advertising contexts, including search, display, social media, mobile, and contextual advertising.Examples include click prediction, CTR estimation, factorization machines, neural networks, and tree-based methods.
  • Publication venues: The article list spans publications from major venues including SIGIR, WWW, KDD, IJCAI, RecSys, NIPS, ICDM, and CIKM.These venues appear alongside the reviewed CTR and click-prediction studies.
  • Model approaches: The listed articles include methods based on logistic regression, factorization machines, gradient boosting, recurrent neural networks, convolutional networks, and deep CTR models.The bibliography includes works such as “Predicting clicks,” “Field-aware Factorization Machines,” “XGBoost,” “Estimating Ads’ Click through Rate with Recurrent Neural Network,” and “Deep CTR Prediction.”

A.4 Datasets for Advertising CTR Prediction

The review summarizes datasets used for advertising CTR prediction, ranging from real-time-bidding and search logs to very large display-advertising datasets.

  • Real-time bidding data: The iPinYou dataset contains advertising bid, impression, click, and conversion data for real-time bidding and CTR prediction.It originated from the 2013 Global RTB Bidding Algorithm Competition.
  • Large-scale click logs: The Outbrain dataset contains user page views and clicks observed across multiple United States publisher sites from 14-June-2016 to 28-June-2016.Its page-views log exceeds 2 billion rows and 100GB uncompressed.
  • Display advertising data: The Aliyun Taobao dataset contains 26 million display-advertising records sampled from 1.1 million users over eight days.The records are click logs from an online display-advertising platform operated by Alimama Inc.
  • Class-imbalanced data: The Huawei DIGX dataset contains 159,837,655 samples with an approximately 1:15 positive-to-negative ratio over six consecutive days.The dataset was created for the 2019 Huawei DIGX algorithm contest.
Loading 2202.10462v1…