Source-linked AI summary
CaliCausalRank: Calibrated Multi-Objective Ad Ranking with Robust Counterfactual Utility Optimization
Xikai Yang, Sebastian Sun, Yilin Li, Yue Xing, Ming Wang, Yang Wang
TL;DR
Ad ranking must reconcile multiple objectives while addressing inconsistent score scales and biased click logs. CaliCausalRank combines training-time calibration, constraint-based optimization, and robust counterfactual utility estimation, achieving consistent benchmark gains over PairRank while retaining stated scope limitations.
Problem
Ad ranking faces inconsistent score scales across traffic segments and position-biased click logs that can undermine threshold transferability and offline-online metric agreement.
Method
CaliCausalRank jointly uses training-time scale calibration, Lagrangian constraint optimization, and variance-reduced counterfactual utility estimation.
Results
1.1% relative AUC improvement, 31.6% ECE reduction, and 3.2% Utility@10 gain are reported versus PairRank across the benchmark evaluation.
Takeaways & Limitations
The reported results indicate moderate, consistent improvements in ranking quality, calibration error, and counterfactual utility on public benchmarks.
Takeaways & Limitations
The experiments use public CTR datasets with synthesized CVR labels and simulated CPC and risk constraints rather than proprietary data with real conversion signals.
Abstract
from arXiv · showhide
Ad ranking systems must simultaneously optimize multiple objectives including click-through rate (CTR), conversion rate (CVR), revenue, and user experience metrics. However, production systems face critical challenges: score scale inconsistency across traffic segments undermines threshold transferability, and position bias in click logs causes offline-online metric discrepancies. We propose CaliCausalRank, a unified framework that integrates training-time scale calibration, constraint-based multi-objective optimization, and robust counterfactual utility estimation. Our approach treats score calibration as a first-class training objective rather than post-hoc processing, employs Lagrangian relaxation for constraint satisfaction, and utilizes variance-reduced counterfactual estimators for reliable offline evaluation. Experiments on the Criteo and Avazu datasets demonstrate that CaliCausalRank achieves 1.1% relative AUC improvement, 31.6% calibration error reduction, and 3.2% utility gain compared to the best baseline (PairRank) while maintaining consistent performance across different traffic segments.
I. INTRODUCTION
CaliCausalRank targets multi-objective ad ranking challenges involving inconsistent score scales, biased click logs, and hard operational constraints. It unifies training-time calibration, constraint-aware optimization, and robust counterfactual estimation to address these challenges jointly.
- Ad ranking must balance CTR, CVR, revenue, CPC and ROAS targets, and user-experience metrics such as bounce rate and complaint frequency.
- Weighted multi-objective scores can have inconsistent scales across traffic segments, hindering threshold and policy transfer.
- Biased click logs create position and selection bias, producing cases where offline metrics improve while online performance degrades.
- CaliCausalRank jointly integrates training-time scale calibration, Lagrangian constraint optimization, and variance-reduced counterfactual utility estimation.The framework treats calibration and bias correction as components of an end-to-end procedure rather than isolated or post-processing steps.
- The framework draws on counterfactual learning, constrained multi-objective optimization, and structurally consistent representation learning with uncertainty control.
- Related methods motivate importance weighting for biased feedback, variance-aware utility estimation, constraint-aware optimization, and alignment across heterogeneous objectives.
A. Problem Formulation
CaliCausalRank formulates ad ranking as expected-utility maximization under operational constraints, using shared representations, task-specific predictions, and calibrated scoring components.
- A. Problem Formulation: The ranking objective is to maximize expected utility for a user and candidate-ad set while satisfying operational constraints.
- A. Problem Formulation: The model predicts a relevance score representing click probability.
- A. Problem Formulation: The model predicts a revenue score representing expected revenue contribution.
- A. Problem Formulation: The model predicts a risk score representing the probability of negative outcomes.
- A. Problem Formulation: The final ranking score combines task predictions subject to CPC, risk-tolerance, and advertiser-fairness constraints.
- B. Model Architecture: The architecture contains shared feature-extraction layers, task-specific prediction heads, and a scale-calibration module.
- B. Model Architecture: Shared layers encode user-ad interactions using deep crossing networks and attention mechanisms.
- B. Model Architecture: Each task-specific tower outputs a raw score for the relevance, revenue, or risk task.
C. Training-Time Scale Calibration
The calibration module makes score-scale consistency a differentiable training objective by matching predictions to empirical rates within contextual buckets.
- The calibration module partitions training data into K contextual buckets, such as traffic source or device type.
- Its calibration loss enforces agreement between predicted scores and empirical rates within each bucket.
- The L1 formulation uses each bucket's predicted score and empirical rate as its core quantities.
- The calibration loss serves as a differentiable surrogate for bin-wise calibration error and approximates the ECE objective during training.
D. Constraint-Based Multi-Objective Optimization
CaliCausalRank formulates ranking as constrained utility maximization and uses Lagrangian relaxation to balance utility against CPC, risk, and fairness requirements.
- The method replaces manual multi-objective weight tuning with a constrained optimization formulation.
- Fairness constraints regulate advertiser exposure.
- The utility function can represent expected revenue, while CPC and risk are bounded by maximum thresholds.
- Because public datasets lack auction logs, CPC uses synthetic log-normal bids and risk uses predicted low-engagement probability in a proxy setting.
- Lagrangian relaxation incorporates operational constraints into the optimization objective.
- Dual gradient ascent updates the CPC and risk multipliers during training to balance utility maximization and constraint satisfaction.
E. Robust Counterfactual Utility Estimation
CaliCausalRank models position bias through examination probabilities and uses self-normalized importance sampling to reduce counterfactual-estimation variance. Its utility loss includes variance regularization to control the bias-variance trade-off.
- Position-bias modeling: Position bias is modeled by decomposing click probability into examination and relevance-conditioned click probabilities.The framework uses randomized impressions to estimate position-dependent examination propensities.
- Variance reduction: Self-normalized importance sampling is used to reduce the high variance of standard IPS estimators.The estimator uses reward and estimated examination probability terms in counterfactual utility estimation.
- Utility optimization: The counterfactual utility loss combines estimated utility with a variance regularization term.The regularization coefficient controls the bias-variance trade-off.
IV. EXPERIMENTS
Experiments use the public Criteo and Avazu benchmarks, synthetic multi-task CVR labels, standard ranking and calibration metrics, and specified PyTorch model settings. The CVR construction is explicitly acknowledged as a proxy for real-world evaluation.
- Datasets: Criteo contains 45M samples, while Avazu contains 40M mobile ad impressions with device and temporal features.Criteo uses 13 numerical and 26 categorical features for CTR prediction.
- Datasets: Synthetic CVR labels are generated only for clicked samples, with non-clicked samples assigned CVR=0 and excluded from CVR loss computation.The setup uses a 10% base conversion probability weighted by feature embeddings and is intended to avoid label leakage.
- Baselines: Baselines include DeepFM, MMoE, PLE, ESCM, and PairRank.PairRank uses pairwise ranking with off-policy correction.
- Metrics: Evaluation reports AUC, NDCG@10, Expected Calibration Error, and counterfactual Utility@10.These metrics cover ranking quality, top-k performance, calibration, and utility.
- Implementation: Models use PyTorch with embedding dimension 16, hidden layers [256, 128, 64], and Adam optimization.The implementation also specifies calibration buckets, variance regularization, and loss weights.
B. Main Results
CaliCausalRank outperforms the reported baselines on the Criteo and Avazu comparisons, with gains in ranking, calibration, and counterfactual utility. The strongest reported calibration and utility improvements are measured against PairRank.
- Main comparison: 1.1% relative AUC improvement is achieved on Criteo over PairRank, with scores of 0.7842 versus 0.7756.The passage identifies PairRank as the best baseline for this comparison.
- Main comparison: 31.6% lower Expected Calibration Error is achieved on Criteo than PairRank, with values of 0.0312 versus 0.0456.The result is reported as evidence for training-time calibration.
- Main comparison: The comparison table reports CaliCausalRank as consistently outperforming all baselines across both datasets.The table is identified as the main comparison of performance on Criteo and Avazu.
- Main comparison: 3.2% Utility@10 improvement over PairRank is reported on both Criteo and Avazu.The reported values are 0.7156 versus 0.6934 on Criteo and 0.7234 versus 0.7012 on Avazu.
C. Ablation Study
The ablation results attribute complementary benefits to calibration, counterfactual estimation, and constraint handling. The full model achieves the best performance across all reported metrics on Criteo.
- Component ablations: Removing calibration increases ECE by 88.1%, from 0.0312 to 0.0587, and reduces utility by 4.5%.This ablation examines the contribution of the calibration module.
- Component ablations: Removing counterfactual estimation decreases AUC by 1.9% and utility by 9.0%.The result is reported as evidence that bias correction is important for reliable training.
- Component ablations: Removing the constraint module reduces utility by 3.3%.The passage associates this module with respecting operational bounds during optimization.
- Component ablations: The full model achieves the best performance across all metrics in the Criteo ablation study.Rows represent model variants, and colors indicate relative performance.
E. Cross-Domain Transferability
CaliCausalRank is designed to support ranking-policy transfer across traffic segments through training-time scale calibration. In a proxy cross-domain evaluation, it retained more of its original AUC than ESCM and PLE, while the authors note that the setup may not fully represent real cross-device transfer.
- Transferability: CaliCausalRank retained 94.2% ±1.3% of its original AUC when trained on desktop-like traffic and evaluated on mobile-like traffic without retraining.Its absolute AUC was 0.739 ± 0.010 versus 0.784 originally, across 5 random splits.
- Transferability: 94.2% ±1.3% compared with 87.6% ±2.1% for 2 ESCM and 82.3% ±2.8% for PLE in the cross-segment evaluation.The corresponding absolute AUCs were 0.674 ± 0.016 for 2 ESCM and 0.619 ± 0.021 for PLE.
- Mechanism: Training-time scale calibration is intended to make ranking policies transferable across domains by enforcing score consistency during training.The framework treats calibration as one component of a unified approach alongside constraint optimization and counterfactual utility estimation.
- Overall results: The broader benchmark results report 1.1% AUC improvement, 31.6% ECE reduction, and 3.2% utility gain compared with the best baseline.These results are reported as moderate but consistent improvements across public benchmarks.
- Caveat: The cross-domain experiment uses feature-based clustering as a proxy for device type and may not fully reflect real cross-device scenarios.The authors identify validation on production datasets and more realistic transfer settings as future work.