Real-Time Bidding Benchmarking with iPinYou Dataset

1. INTRODUCTION

• DSPs receive bid requests through ad exchanges and calculate a bid for each impression, typically within a very short response window such as 100 ms.Source: §1 · p. 1
• DSP algorithms must decide quickly whether and how much to bid using contextual and behavioral data, amid potentially billion-level daily bid requests.Source: §1 · p. 1
• The iPinYou competition dataset was released for research, providing a large-scale real-world RTB dataset when academic access to sensitive industry data was nearly impossible.The dataset was about 35 GB and the task focused on maximizing a campaign KPI subject to budget and lifetime constraints.Source: §1 · p. 1
• The paper analyzes the dataset, formally presents DSP bid optimisation and an evaluation protocol, and reports benchmark experiments for bidding strategies and CTR estimation.Source: §1 · p. 2

2.1 iPinYou Demand-Side Platform

• iPinYou is a China-based demand-side platform whose dataset includes advertisers from diverse industries, with user-response behavior varying across fields.Source: §2.1 · p. 2, §2.3 · p. 3

2.2 Data Format

• The dataset contains four row-per-record log types—bids, impressions, clicks, and conversions—with auction features, winning prices, and user feedback represented in the ad-log data.Source: §2.2 · p. 2, Table 1 · p. 2

2.3 Basic Statistics

• CTR is below 0.1% for every advertiser except 2997 (0.444%); eCPC and conversion rates also vary, and some advertisers report no conversions.Source: §2.3 · p. 3, Table 3 · p. 4

2.4 User Feedback

• The same features can have different CTR patterns across advertisers: mobile users are more likely to click advertiser 1458’s ads, while PC users prefer advertiser 3358’s.Source: §2.4 · pp. 3–4, Figure 2 · p. 3

2.5 Bidding Behaviour

• Market-price patterns differ by advertiser: advertiser 1458 has higher morning competitiveness, while advertiser 3358 has higher afternoon and evening competitiveness.Source: §2.5 · p. 4, Figure 3 · p. 5

2.6 eCPC

• For advertiser 3358, eCPC varies across features: Sunday is costliest, Monday and Wednesday are more cost-effective, and iOS users are about 3 times as cost-effective as Windows users.Source: §2.6 · p. 5, Figure 4 · p. 5
• Because feature effects on user feedback and market price differ across advertisers, the authors consider it reasonable to build user-response models and bidding strategies independently for each advertiser.Source: §2.4 · p. 4, §2.6 · p. 5

3.1 Task Description

• The bidding task optimizes an advertiser’s predefined KPI under a cost budget and incoming requests; the competition KPI combines clicks with N-weighted conversions.N represents the relative importance of conversions to clicks and varies by advertiser; conversions are a final advertising measure but are sparse.Source: §3.1 · pp. 5–6

3.2 Training Framework

• Training uses bid-request features and user-feedback labels to estimate CTR, combines that estimate with a market-price model, and optimizes a bidding function.The price distribution can estimate auction-win probability and cost for a bid; the framework does not prescribe one bidding-function optimization process.Source: §3.2 · p. 6

3.3 Evaluation Protocol

• Offline evaluation replays timestamp-ordered bid requests, simulates wins against logged prices, matches feedback for won impressions, and updates performance and cost until the test data ends.Bids are set to zero after the budget is exhausted; a tiny overspend from the last impression is neglected.Source: §3.3 · pp. 6–7
• Because logged click and conversion feedback exists only for won auctions, replay cannot test whether bidding higher would improve performance on lost auctions.The authors follow offline-evaluation conventions that ignore objects with unseen user feedback.Source: §3.3 · p. 7

• The benchmark follows a two-stage RTB process: estimate CTR or CVR for an impression, then make a bid decision using that estimate and other information.Source: §4 · p. 7
• Because more than half of advertisers have no conversion record and conversion settings differ, the benchmark focuses on clicks.Source: §4 · p. 7
• The experiments compare two standard CTR estimators and several pCTR-based bidding strategies, considering clicks, conversions, and the iPinYou KPI.Source: §4 · p. 7

4.1.1 Compared Models and Feature Engineering

• LR uses binary features and cross-entropy loss with L2 regularization, while GBRT uses continuous features and can learn nonlinear features.The experiment uses 937,748 binary LR features and 78 GBRT features; GBRT is implemented with xgboost, depth 5, 50 trees, and a 0.05 learning rate.Source: §4.1.1 · p. 7

4.1.3 Results

• Both LR and GBRT perform better on season 2 advertisers than on season 3 advertisers, while CTR-estimation AUC and RMSE vary substantially across advertisers.The paper attributes the season difference, as explained by iPinYou technicians, to different user segmentation systems.Source: Table 4 · p. 7, §4.1.3 · pp. 7–8

4.1.2 Evaluation Measure

• AUC and RMSE are the evaluation measures; severe click-class imbalance keeps empirically strong pCTR predictions close to zero, making RMSE improvements comparatively slight.Source: §4.1.2 · p. 7

4.1.3 Results

• Advertiser 2997 has the highest overall CTR (0.444%) but the fewest observations, making its CTR harder to predict.Source: §4.1.3 · pp. 7–8

4.2.1 Compared Bidding Strategies

• The benchmark tests Const, Rand, Mcpc, and linear-form pCTR bidding, with LR or GBRT CTR estimators for Mcpc and Lin.Mcpc is non-parametric and does not consider budget limits; the other strategies tune parameters, and Mcpc and Lin require impression-level CTR estimates.Source: §4.2.1 · p. 8

4.2.2 Experimental Setting

• Each advertiser’s budget is set to 1/32, 1/8, or 1/2 of its original total test-log cost.The evaluation uses the protocol described in Section 3.3.Source: §4.2.2 · p. 8

4.2.3 Results

• Lin and Mcpc generally achieve more clicks than Const and Rand, but Mcpc can underperform them for some advertisers at low budgets.Lin adapts its bidding scale through base_bid; Mcpc does not adapt to different budget limits.Source: §4.2.3 · p. 8, Table 5 · p. 8
• Lin significantly improves conversion and KPI performance over other strategies, with higher improvement rates at lower budgets.The KPI is #click + N · #conversion.Source: §4.2.3 · pp. 8–9, Table 6 · p. 9

5. OTHER RESEARCH TOPICS

• The dataset can support experimentation beyond CTR estimation and DSP bid optimisation.The paper names bid landscape modelling and adaptive bid control as examples.Source: §5 · p. 9
• Bid landscape modelling targets market-price distributions, which matter because optimal bidding in repeated auctions with budget constraints depends on them.The passage notes that current DSP work mainly bases bidding on CTR/CVR estimates.Source: §5 · p. 9
• Adaptive bid control targets pacing bids to deliver the budget across a campaign’s lifetime.A high bidding scale early in a campaign can exhaust the budget at the beginning; the passage points to pacing and MDP-based approaches as relevant work.Source: §5 · p. 9

6. CONCLUSIONS

• The authors expect the iPinYou dataset’s publication to stimulate RTB display-advertising research, which they say is restricted by data availability.Source: §6 · p. 9
• The paper analyzes the dataset, defines bid optimisation, presents an offline evaluation protocol, and benchmarks CTR estimation and DSP bid optimisation.The dataset and report are hosted on the computational advertising research website.Source: §6 · p. 9