PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond IPS: Reliable Counterfactual Evaluation in Multi-Stage Ad Systems without Logged Propensities

Mohsen Malmir, Mohamed A. Radwan, Houssam Nassif, Murat Ali Bayir

arXiv:2610.40112v1cs.IR

TL;DR

Offline counterfactual evaluation in multi-stage ad systems is difficult because propensities may be unavailable and reward models can be biased by distribution shift. The paper reweights reward-model training toward candidate-policy mixtures and analyzes replay error and routing. The method improves policy selection in simulation and production, while its routing rule remains asymptotic.

  • Problem

    Multi-stage ad systems may lack logged action propensities, and reward models can be biased when candidate-policy impressions differ from the logged distribution.

    Source: §3.2 · p. 4

  • Method

    The paper trains a reward model on logged data reweighted toward a mixture of candidate-policy distributions using density ratios from a multiclass domain classifier.

    Source: §1 · p. 2, §6 · p. 10

  • Results

    Mixture-weighted reward modeling improves policy selection over its unweighted counterpart in simulation and production, with the largest gains at low overlap; with a doubly robust correction, it achieves the highest Top-1 accuracy at low and medium overlap and on production data.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    The analysis uses cold-start mass to inform whether offline selection is trustworthy or candidates should be routed to online A/B testing.

    Source: §6 · p. 10

    The routing rule is asymptotic, and its operational performance under live traffic is left to future work.

    Source: §3.5 · p. 7, §6 · p. 10

Abstract

from arXiv · show

Offline evaluation of ad-ranking policies in multi-stage delivery systems is challenging because action propensities are often unavailable, limiting inverse propensity scoring (IPS). Reward-model-based estimators such as the Direct Method (DM) avoid propensities but can be biased when candidate policies induce impression distributions that differ from the logged data. We formalize DM-based counterfactual replay and propose mixture-weighted reward modeling, which reweights logged data toward a mixture of candidate-policy distributions using density ratios from a multiclass domain classifier. We bound each candidate's value-estimation error by overlap, cold-start, and replay terms, and derive an asymptotic rule that uses the cold-start mass to decide whether to trust offline selection or route candidates to online A/B testing. On an ads-delivery simulator and on production traffic from a large-scale CTR ranking system, the mixture-weighted reward model improves policy selection over its unweighted counterpart in both settings, with the largest gains at low overlap in simulation. Combined with a doubly robust correction, it attains the highest Top-1 accuracy in production and at low and medium overlap in simulation.

1 Introduction

Multi-stage ad delivery makes offline counterfactual evaluation difficult: propensities may be unavailable, and reward models can be biased under policy-induced distribution shift. The paper proposes mixture-weighted reward modeling and analyzes when offline policy selection is reliable.

1 Introduction

  • Action propensities are often unavailable in multi-stage ad systems, making inverse propensity estimators inapplicable while logged data reflect a changing mixture of production policies.Exposure depends on multiple ranking models and platform operations, rather than a single stochastic ranking policy.Source: §1 · pp. 1–2
  • Predictive metrics measure logged-data accuracy rather than the counterfactual value of a new policy, while IPS requires propensities that post-ranking operations can make unavailable.Replay with learned response models faces a different difficulty: predicting outcomes for unlogged actions, often under limited overlap.Source: §1 · p. 2
  • Reward models replay candidate policies on historical contexts, but distribution shift can make predictions unreliable on target-policy actions and bias estimated policy values.The risk arises when candidate-action distributions differ substantially from the logging mixture.Source: §1 · p. 2
  • Mixture-weighted reward modeling improves policy selection over its unweighted counterpart in simulation and production, with the largest gains at low overlap.The paper also reports that a doubly robust correction achieves the highest Top-1 accuracy at low and medium overlap in simulation and on production data.Source: §1 · p. 2
  • The proposed framework reweights training toward a mixture of candidate-policy distributions and bounds value error using overlap, cold-start, and finite-replay terms.Its asymptotic rule uses cold-start mass to inform whether offline selection is sufficient or candidates should go to online A/B testing.Source: §1 · p. 2

2 Literature Review

Prior approaches evaluate ad ranking using logged-data prediction metrics, off-policy estimators, or auction and delivery simulation. Each has limits when the goal is to estimate a new policy’s counterfactual value in a multi-stage system.

2 Literature Review

  • Logged-impression predictive metrics such as log-loss, AUROC, AUC-lift, and NDCG measure performance under production exposure but not counterfactual effects on delivery, user experience, or revenue.The review gives examples including logistic regression, Wide & Deep [Cheng et al., 2016], and ESMM [Ma et al., 2018].Source: §2 · p. 2
  • IPS and doubly robust off-policy estimators require known propensities, target-policy support, and no unobserved transformations between action selection and outcome.Multi-stage operations such as auctions and audience targeting can transform rankings in unlogged ways, limiting value identification.Source: §2 · pp. 2–3
  • Auction replay and simulation evaluate policy changes by modeling auctions and user responses, with high fidelity depending on downstream operations being fully specified and stable.Examples include auction replay and simulation systems [Graepel et al., 2010; Yi et al., 2013; Bottou et al., 2013; Bayir et al., 2019; Bai et al., 2018; Jeunen et al., 2023; Yin, 2023].Source: §2 · p. 3

3 Approach

The approach evaluates candidate policies by replaying their actions on logged contexts, then reweights reward-model training toward a mixture of candidate-policy distributions. It uses domain-classifier ratios to estimate weights and bounds replay-DM error to guide offline selection versus A/B testing.

3.1 Problem Setup

  • Candidate policies are evaluated on logged contexts to estimate their values and rank them for scarce A/B-testing slots.The analysis assumes candidate actions can be replayed on logged contexts and that the reward distribution for an impression is unchanged across logging and candidate policies.Source: §3.1 · p. 3

3.2 Direct Method based Approach

  • Without reliable propensities, Direct Method replay predicts rewards for each candidate action on logged contexts and averages predictions over held-out contexts.The paper uses the empirical replay mean on a held-out evaluation split.Source: §3.2 · p. 4

3.3 Mixture-Weighted Reward Model Training

  • A shared reward model is trained with density-ratio weights to target a mixture of candidate-policy distributions rather than the logging distribution alone.With no prior information about candidate importance, the method uses uniform mixture weights.Source: §3.3 · p. 4

3.4 Estimating Density Ratios via Domain Classification

  • A multiclass domain classifier distinguishes logged impressions from replayed candidate actions to estimate policy density ratios and mixture weights.The classifier labels each impression by its generating policy and is trained with uniform class priors.Source: §3.4 · pp. 4–5

3.5 Value Estimation Error Decomposition

  • The error bound separates overlap, cold-start, and replay terms; asymptotically, cold-start mass guides offline trust versus A/B routing, while finite samples require the full bound.Threshold τ trades lower cold-start mass against a larger 1/τ factor.Source: §3.5 · pp. 5, 7

4 Synthetic Data Experiments

The synthetic experiments compare six estimators across overlap regimes using a simulator with oracle policy values. Mixture weighting improves DM policy selection, while the best DR variant depends on overlap.

4.1 Simulation Environment and Setup

  • The simulator uses 10,000 users, 1,000 ads, 10 candidate ads per impression, and 70,000 logged impressions, with oracle rewards providing ground-truth policy values.Impressions are split into folds for classifier training, reward-model training, calibration, and evaluation; results aggregate 50 seeds.Source: §4.1 · p. 8

4.2 Estimators

  • Weighted DM trains one reward model with mixture-derived weights, whereas Per-policy Weighted DM trains a separate model for each candidate; Weighted DR adds a residual correction to the weighted model.IPS and DR use classifier-based density-ratio estimates because true propensities are unavailable.Source: §4.2 · p. 8

4.3 Evaluation Metrics

  • Top-1 regret measures value lost by selecting a policy other than the oracle-best, Top-1 accuracy measures correct selection, and Value MAE% measures policy-value estimation error.The metrics are averaged over trials and candidate policies as defined in the experiment.Source: §4.3 · p. 8

4.4 Experiment 1: Estimator Comparison Under Distribution Shift

  • Weighted DR leads all three metrics in the Low and Mid regimes; in High, Unweighted DR has slightly higher Top-1 accuracy, regret ties, and Weighted DR has lower Value MAE%.Weighted DM improves over Unweighted DM on eight of nine regime-metric comparisons and beats Per-policy Weighted DM on all nine.Source: Table 1 · p. 9, §4.4 · p. 9

5 Production Data Experiment

On production traffic, Weighted DM leads the DM estimators across all three metrics, while DR methods achieve lower value error but do not dominate on policy selection. The near-tied top candidates remain difficult to distinguish offline.

Setup

  • The production evaluation uses one logging policy and five candidate treatments, each served on a separate share of live traffic, with metrics averaged over 50 random subsamples.The candidates' observed click-through rates provide their true values for evaluation.Source: §5 · p. 9

Results

  • 0.40 Top-1 accuracy, 0.00244 regret, and 11.56 Value MAE% make Weighted DM best among DM estimators, beating Per-policy Weighted DM (0.24, 0.00676, 12.43).Weighted DM also beats Unweighted DM, which records 0.36 Top-1 accuracy, 0.00271 regret, and 14.10 Value MAE%.Source: §5 · p. 9, Table 2 · p. 10
  • Weighted DR achieves the highest Top-1 accuracy among DR estimators, while Unweighted DR has marginally lower regret and Value MAE%, so neither DR variant dominates.IPS and both DR estimators have Value MAE% of 1.90–4.14, versus 11.56–14.10 for DM estimators; IPS has 0.00807 regret and 0.24 Top-1 accuracy.Source: §5 · p. 10
  • The estimated production winner exceeds the runner-up by only 0.15% on average, and their ordering reverses in 24 of 50 subsamples, so the candidates should go to online A/B testing.The paired-bootstrap 95% interval for the winner's margin is [−0.31%, 0.60%].Source: §5 · p. 10

6 Conclusions

The paper concludes that mixture-weighted reward modeling improves DM replay under policy-induced shift, and that overlap and cold-start diagnostics can guide whether to trust offline selection or use online testing.

6 Conclusions

  • The replay-DM error bound separates overlap, cold-start, and replay-concentration terms, and an asymptotic rule uses cold-start mass to route candidates between offline selection and online A/B testing.The routing choice is framed for a given regret budget.Source: §6 · p. 10
  • Mixture-weighted DM improves policy selection most at weak overlap, while its doubly robust correction yields the highest Top-1 accuracy in production and at low and medium overlap in simulation.The value-error bound retains a cold-start term for impressions unsupported by logging data, which more logged data does not reduce.Source: §6 · p. 10
  • The analysis assumes replayable candidate policies and policy-independent impression reward distributions; the routing rule is asymptotic and its live-traffic performance remains future work.With fixed clipping, vanishing weight error requires the mixture ratio never to exceed the clip.Source: §6 · p. 10

A Synthetic Simulator Details

  • The synthetic simulator specifies user and ad features, an oracle reward function, logging and candidate policies, and a per-trial evaluation pipeline.The paper reports public release of the simulator and synthetic evaluation code.Source: A Synthetic Simulator Details · p. 13

A.1 Users, ads, and impression generation

  • Each simulated impression samples a user and ten candidate ads from fixed corpora, selects the highest-scoring ad, and assigns its oracle expected reward.The corpora contain 10,000 users and 1,000 ads, and each trial uses 70,000 impressions.Source: §A.1 · p. 13

A.2 Oracle reward function

  • The oracle reward is a sum of 400 Gaussian kernels over joint user–ad features, with each product kernel truncated at 1.The oracle is redrawn per seed, and candidate policies in a trial share that ground truth.Source: §A.2 · p. 13

A.3 Logging policy and candidate-policy parameterization

  • The logging and candidate policies use Gaussian-kernel scores, with candidate parameters selected by a calibration sweep over value multipliers and effective sample size targets.Candidates are omitted when no grid point qualifies; 56 of 350 trials evaluate fewer than six candidates.Source: §A.3 · p. 13

A.4 Per-trial procedure

  • Each trial splits impressions across classifier training, reward-model training, calibration, and evaluation; calibrated classifier outputs produce clipped mixture and per-policy weights.The reward models share architecture and training settings, differing in whether their sample weights are uniform, mixture-based, or policy-specific.Source: §A.4 · pp. 13–14

B Proof of Theorem 3.3: Replay-DM Value Estimation Error

The proof combines a replay concentration bound with weighted-risk and population-bias bounds to establish the candidate value-error guarantee with joint probability control.

Replay bound

  • The replay estimate concentrates around its population mean by Hoeffding's inequality, with a union bound covering all candidate policies.The replay contexts are i.i.d. and independent of the training folds under the stated assumption.Source: B Proof of Theorem 3.3: Replay-DM… · p. 14

Weighted-risk bound

  • The weighted-risk argument uses the mixture density lower bound and support-restricted mixture identity, with calibration and a shared conditional reward law supporting the risk identification.The proof applies the mixture bound and Cauchy–Schwarz on the logging-supported region.Source: B Proof of Theorem 3.3: Replay-DM… · p. 15

Bound assembly

  • Combining the weighted-risk and replay bounds yields the stated error bound, with joint probability at least 1 − δtr − δev.The proof obtains this probability by a final union bound over the training and evaluation events.Source: B Proof of Theorem 3.3: Replay-DM… · p. 15

C Proof of Corollary 3.5: Top-1 Selection Regret

The proof bounds the regret of selecting a candidate by twice the largest candidate value-estimation error.

C Proof of Corollary 3.5: Top-1 Selection Regret

D Proof of Corollary 3.6: Asymptotic Offline-vs-A/B Routing Rule

The proof establishes that the selection regret converges in probability under the stated limits, and that condition (23) makes the probability of exceeding Δ + η vanish.

D Proof of Corollary 3.6: Asymptotic Offline-vs-A/B Routing Rule

E Detailed Per-ESS-Bin Results for Experiment 1

Across realized-ESS bins, Weighted DR has the lowest Value MAE% in every row, while Top-1 accuracy improves overall with overlap and its leading estimator varies by bin.

E Detailed Per-ESS-Bin Results for Experiment 1

F Synthetic Experiment 2: Sensitivity to Importance Weight Clipping

The clipping comparison shows that moderate-to-large thresholds perform similarly, while the best setting depends on the overlap level and metric.

F Synthetic Experiment 2: Sensitivity to Importance Weight Clipping

  • The experiment varies the clip threshold across policy-logging overlap, using kernel-based mixture weights and ESS as the overlap measure.The true mixture density ratio is computed analytically for the simulator’s kernel scoring policies.Source: F Synthetic Experiment 2: Sensitivity… · p. 17
  • At kernel ESS 0.2–0.4, c = 10, 20, and 100 reduce Value MAE% by 4.1–8.2 points versus the unweighted model, and differ by at most 0.52 points across bins.These settings are largely insensitive to the clipping threshold in the reported comparison.Source: F Synthetic Experiment 2: Sensitivity… · p. 17
  • At kernel ESS 0.2–0.4, c = 5 has higher Value MAE% than c = 10, 20, or 100; at ESS 0.5 and above, it matches or slightly improves on them.Source: F Synthetic Experiment 2: Sensitivity… · p. 17
  • At kernel ESS 0.9–1.0, c = 1 yields Value MAE% of 1.54 and 1.38, versus 1.90–2.36 for the unweighted model and other clip values.At kernel ESS 0.3–0.4, c = 1 is close to the unweighted model on Value MAE%, Top-1 regret, and accuracy.Source: F Synthetic Experiment 2: Sensitivity… · p. 17

G Domain Classifier Reliability

The domain-classifier reliability tests find that calibration has little downstream effect, while masking ad features degrades estimator performance. Classifier diagnostics track this degradation, but a high estimated ESS can reflect flattened weights rather than genuine overlap.

G Domain Classifier Reliability

  • Masking ad features lowers Top-1 accuracy for Weighted DM from 0.90 to 0.86, 0.83, and 0.84 at 33%, 67%, and 100% masking, respectively.Weighted DR falls from 0.94 to 0.86, 0.81, and 0.84; IPS falls from 0.62 to 0.27.Source: G Domain Classifier Reliability · p. 18
  • Removing calibration leaves downstream metrics about the same, whereas masking ad features degrades them across estimators.With all ad features masked, weighted estimators end close to their unweighted counterparts.Source: G Domain Classifier Reliability · pp. 17–18
  • At 67% masking, NLL rises from 1.71 to 1.91, candidate accuracy falls from 0.32 to 0.18, ratio correlation falls from 0.47 to 0.24, and cold-start error rises from 0.16 to 0.39.At 100% masking, ratio correlation is undefined because estimated weights are constant.Source: G Domain Classifier Reliability · pp. 18–19
  • Estimated ESS rises from 0.59 to 1.00 under masking because weights flatten, so high ESS indicates good overlap only when the classifier distinguishes policies.NLL and candidate accuracy flag classifier failure without oracle information, but cannot distinguish genuinely similar policies from missing features; simulation's ratio correlation and cold-start error can separate them.Source: G Domain Classifier Reliability · p. 19

H Production Data: Per-Policy Breakdown

This section reports production-data details, including per-policy comparisons of value-estimation errors across DM, IPS, and DR estimators, alongside disclosures about the analysis and study methods.

H Production Data: Per-Policy Breakdown

  • IPS and both DR estimators have lower Value MAE% than every DM estimator on every candidate; IPS ranges from 1.87 on P2 and P4 to 6.87–7.31 on P3, P5, and P6.Unweighted DR ranges from 1.90 to 2.48 and Weighted DR from 2.02 to 2.81.Source: H Production Data: Per-Policy Breakdown · p. 19, Table 9 · p. 20
  • On logging policy P1, IPS and both DR estimators have zero error by construction, while Per-policy Weighted DM coincides with Unweighted DM at 14.40.P1’s density ratio is identically one, so IPS and both DR estimators reduce to the mean of logged rewards.Source: H Production Data: Per-Policy Breakdown · p. 19, Table 9 · p. 20
  • Weighted DM achieves Value MAE% of 10.91–12.24 versus 13.30–15.19 for Unweighted DM on every candidate, and beats Per-policy Weighted DM on four of five candidates.The exception is P6, where Weighted DM is 10.91 versus 10.51 for Per-policy Weighted DM; values use clip c = 20 and average 50 random subsamples.Source: H Production Data: Per-Policy Breakdown · p. 19, Table 9 · p. 20

NeurIPS Paper Checklist

  • The paper’s stated contributions include propensity-free mixture-weighted reward modeling, a replay Direct Method error bound, and an asymptotic offline-versus-A/B routing rule; limitations include the replay assumption and no live-traffic evaluation.The limitations also include dependence on domain-classifier signal and the need for calibration or error correction when estimating cold-start mass.Source: NeurIPS Paper Checklist · p. 21
  • Production data, embeddings, and trained models will not be released, while the simulator and synthetic evaluation code are publicly released; Table 9 reports means and standard errors over 50 production subsamples.The checklist states that the production evaluation uses one logging and five candidate policies with clip c = 20.Source: NeurIPS Paper Checklist · pp. 22–24
  • The paper marks human-subject and crowdsourcing disclosures N/A and states that no IRB or equivalent approval was required.The justification is that the research did not involve human subjects.Source: NeurIPS Paper Checklist · pp. 27–28
  • The production experiment analyzed pre-existing aggregated impression logs, with no new human-subject data collection.The paper states that its synthetic experiments used a simulator with no real users.Source: NeurIPS Paper Checklist · p. 27
  • The paper marks LLM-use disclosure N/A because LLMs were not an important, original, or non-standard part of its core methods.The proposed estimators are described as statistical methods that do not use LLMs.Source: NeurIPS Paper Checklist · p. 28

Related papers

Loading 2610.40112v1…