Source-linked AI summary
To Model or to Intervene: A Comparison of Counterfactual and Online Learning to Rank from User Interactions
Rolf Jagerman, Harrie Oosterhuis, Maarten de Rijke
TL;DR
Practitioners lacked direct evidence for choosing between counterfactual and online LTR, despite the decision's importance for deployment and user satisfaction. This paper compares both approaches under different bias and noise conditions. It finds their relative performance depends on those conditions, with user-experience trade-offs for deployment.
Problem
Practitioners lacked a direct comparison showing when counterfactual or online LTR methods should be preferred, despite the choice's importance for deployment and user satisfaction.
Source: §1 · pp. 1–2Method
The study directly compares counterfactual and online LTR methods across experimental conditions, including propensity-aware counterfactual methods and periodic deployment.
Source: §1 · p. 2, §5.4 · p. 6Results
Counterfactual LTR performs better with little bias or noise, while online LTR is more robust to selection bias, high position bias, and high interaction noise.
Source: §8 · p. 9Takeaways & Limitations
Practitioners must weigh counterfactual methods' risks under noisy deployment against online LTR's initially worse user experience and potential long-term gains.
Source: §8 · pp. 9–10The comparison uses exact propensities and evaluates only one metric on one dataset.
Source: §8 · p. 10
Abstract
from arXiv · showhide
Learning to Rank (LTR) from user interactions is challenging as user feedback often contains high levels of bias and noise. At the moment, two methodologies for dealing with bias prevail in the field of LTR: counterfactual methods that learn from historical data and model user behavior to deal with biases; and online methods that perform interventions to deal with bias but use no explicit user models. For practitioners the decision between either methodology is very important because of its direct impact on end users. Nevertheless, there has never been a direct comparison between these two approaches to unbiased LTR. In this study we provide the first benchmarking of both counterfactual and online LTR methods under different experimental conditions. Our results show that the choice between the methodologies is consequential and depends on the presence of selection bias, and the degree of position bias and interaction noise. In settings with little bias or noise counterfactual methods can obtain the highest ranking performance; however, in other circumstances their optimization can be detrimental to the user experience. Conversely, online methods are very robust to bias and noise but require control over the displayed rankings. Our findings confirm and contradict existing expectations on the impact of model-based and intervention-based methods in LTR, and allow practitioners to make an informed decision between the two methodologies.
1 INTRODUCTION
The paper directly compares counterfactual and online LTR to help practitioners choose between methods that handle biased, noisy user interactions differently. It finds that the better choice depends on experimental conditions, with consequences for ranking performance and user experience.
1 INTRODUCTION
- User interactions can provide preference-aligned, lower-cost feedback than expert annotations, but their bias and noise motivate methods that remove bias from learning.The introduction notes that interaction-based LTR can also respect privacy-sensitive settings.Source: §1 · p. 1
- Counterfactual methods learn from historical interaction logs without experimental control, whereas online methods intervene by displaying modified rankings to learn from interactions.Online control can reveal novel rankings but risks exposing users to untested rankings that may hurt their experience.Source: §1 · p. 1
- Before this study, no direct comparison established when counterfactual or online methods should be preferred, despite the decision's importance for deployment and user satisfaction.The paper also frames the comparison around convergence, user experience during learning, and the effect of deploying counterfactual models periodically.Source: §1 · pp. 1–2
- Online LTR is more robust to selection bias, position bias, and interaction noise, while counterfactual LTR can perform significantly better with little bias or noise.Counterfactual LTR may also harm user experience under high noise, and online LTR is not substantially faster than counterfactual LTR.Source: §1 · p. 2
2 COUNTERFACTUAL LEARNING TO RANK
Counterfactual LTR learns from historical user interactions, using propensity weighting to address partial feedback and bias. The section describes objectives for rank and DCG optimization and assumes propensities are known.
2 COUNTERFACTUAL LEARNING TO RANK
- Counterfactual LTR learns offline from historical interaction data, avoiding online deployment overhead and allowing models to be tested before deployment.Clicks provide only partial information about relevance, since users do not see every document that could have been clicked.Source: §2 · p. 2
2.1 Unbiased LTR with biased feedback
- Inverse Propensity Scoring reweights clicks by observation propensity; CF-RANK uses the resulting unbiased objective to minimize the average rank of relevant results.The method assumes propensities are known.Source: §2.1 · pp. 2–3
2.2 Unbiased LTR with additive metrics
- CF-DCG adapts counterfactual learning to optimize DCG with a continuous, subdifferentiable objective that can be solved using gradient descent.The paper contrasts this objective with CF-RANK, which attempts to minimize relevant-result rank.Source: §2.2 · p. 3
2.3 Propensity estimation methods
- The comparison assumes propensity scores are known a priori, excluding propensity estimation so results are not confounded by estimator accuracy.The paper notes that practitioners typically measure observation probabilities in a randomization experiment before applying counterfactual learning.Source: §2.3 · p. 3
3 ONLINE LEARNING TO RANK
Online LTR learns through user interactions and uses stochasticity rather than explicit user models to handle position and selection bias. The section contrasts DBGD with PDGD, whose updates use inferred document preferences.
3 ONLINE LEARNING TO RANK
- OLTR learns by interacting with users, using stochasticity to handle position and selection bias without explicitly modeling user behavior.Interventions can affect the data gathered and user experience during learning; unreliable methods may immediately worsen that experience.Source: §3 · p. 3
3.1 Dueling bandit gradient descent
- DBGD compares its current ranking model with a sampled variation through interleaving, then updates toward the variation when users prefer it.Extensions improve DBGD’s initial learning speed, but none is shown to improve long-term performance; DBGD also falls short of labeled-data LTR even under ideal conditions.Source: §3.1 · p. 4
3.2 Pairwise differentiable gradient descent
- PDGD samples rankings from a Plackett-Luce distribution over document scores, then infers pairwise preferences from clicks to estimate a model gradient.The τ parameter controls the initial distribution’s sharpness and reflects confidence in the initial model.Source: §3.2 · p. 4
- PDGD weights pairwise updates to address bias, and its expected gradient is unbiased toward users’ pairwise preferences.Its weighting uses the probability ratio between a ranking and the ranking formed by swapping a document pair.Source: §3.2 · p. 4
4 EXPECTATIONS FROM PREVIOUS WORK
The paper reviews expectations that CLTR and OLTR differ in convergence, user experience during learning, and learning speed when interventions are allowed. It presents the study as the first direct experimental test of these expectations.
Expectation 1 – The performance at convergence
- With correctly known observation probabilities, CLTR can unbiasedly optimize additive metrics such as CF-DCG, while no proof establishes that PDGD unbiasedly optimizes a metric.This motivates the expectation that CLTR could reach higher performance at convergence when propensities are known.Source: §4 · pp. 4–5
Expectation 2 – The user experience during learning
- OLTR is expected to risk a worse initial user experience through noise and exploration, but eventually outperform the unchanged experience under CLTR.CLTR uses historical data and therefore neither changes nor improves the experience during learning.Source: §4 · p. 5
Expectation 3 – The effect of interventions
- Interventions are expected to reduce data requirements and speed learning, with periodic deployment likewise expected to make CLTR more efficient.The expectation is that OLTR needs fewer interactions than CLTR to reach decent performance.Source: §4 · p. 5
Expectations from previous work
- The study presents itself as the first to test these previously untested CLTR–OLTR expectations with experimental evidence.The expectations draw on previously published work.Source: §4 · p. 5
5 EXPERIMENTS
The experiments use Yahoo! Webscope data and simulated user behavior to compare an online method with counterfactual methods across varied bias and noise conditions. They evaluate ranking performance over time and user experience during OLTR learning.
5 EXPERIMENTS
- The experiments extend synthetic LTR settings using a dataset and simulated users, varying interaction noise, position bias, and selection bias.The authors describe this as a comparison scope spanning large- and small-scale LTR requirements.Source: §5 · p. 5
5.1 Optimization setup
- Optimization uses a small-data-trained SVMRank as the CLTR logging ranker and OLTR warm start, then cycles simulated queries, rankings, clicks, and updates.The setup uses 1% of training queries to train the baseline ranker; clicks are generated from displayed rankings and passed to the LTR method.Source: §5.1 · p. 5
5.2 Simulating user behavior
- Ten simulated user types cover three click behaviors, two position-bias levels, and selection bias that hides documents beyond rank 10.Perfect, Binarized, and Near-Random click behaviors differ in relevance sensitivity; the Perfect model has no position bias.Source: §5.2 · pp. 5–6
5.3 Evaluation
- Rankers are evaluated on held-out annotated data using nDCG@10, while OLTR’s learning-time experience is measured with exploration on held-out queries.Results are averaged over multiple runs and compared using a two-tailed t-test, with significance indicated at p < 0.01.Source: §5.3 · p. 6
5.4 Methods and interventions
- The comparison uses PDGD, CF-RANK, and CF-DCG; the counterfactual methods receive actual propensity scores and are also tested with deployment every 200,000 iterations.CF-DCG optimizes the evaluation metric, while CF-RANK is the original counterfactual method.Source: §5.4 · p. 6
6 RESULTS AND ANALYSIS
Online and counterfactual LTR reach different performance levels depending on selection bias, position bias, and interaction noise. Online learning is more robust under difficult conditions, while counterfactual methods can excel with little bias and noise; interventions also trade early user experience against later ranking improvements.
6.1 Ranking performance
- Without selection bias, counterfactual methods are comparable to PDGD under perfect and binarized clicks, but fall below the production ranker under near-random clicks; PDGD is less affected by noise.For binarized clicks, CF-DCG outperforms PDGD (p < 0.01); for near-random clicks, counterfactual methods do not exceed the production ranker within 1,000,000 iterations (p < 0.01).Source: §6.1 · p. 6, Figure 1 · p. 7
- With selection bias, counterfactual performance worsens across user models, while PDGD is little affected and outperforms both counterfactual methods for all click behaviors.The comparison is shown in Figure 1’s bottom row, where a selection of 10 is used.Source: §6.1 · pp. 6–7, Figure 1 · p. 7
- Under very strong position bias (η = 2), PDGD outperforms counterfactual methods in all cases, and online learning handles high position bias better.The authors attribute counterfactual methods’ lower performance to high variance, despite their unbiasedness under the binarized, no-selection-bias condition.Source: §6.1 · p. 7, Figure 2 · p. 7
- Online methods perform best with selection bias, high position bias, or high interaction noise, while counterfactual methods can outperform them when these conditions are mild.The choice of method therefore depends on the bias and noise in the deployment scenario.Source: §6.1 · p. 7, Figure 1 · p. 7, Figure 2 · p. 7
6.2 User experience
- PDGD initially displays worse rankings than the production ranker, but surpasses it after 1,000, 2,000, and 21,000 sessions for perfect, binarized, and near-random clicks, respectively.The comparison concerns displayed ranking quality during training, as shown in Figure 3.Source: Figure 3 · p. 8, §6.2 · p. 8
6.3 The power of interventions
- Periodic deployment of counterfactual models has no negative effect under perfect or binarized clicks and substantially improves performance with selection bias, but worsens performance under near-random clicks.The user experience also improves sooner or deteriorates sooner after deployment; CF-DCG with periodic deployment can outperform PDGD when noise is low (p < 0.01).Source: Figure 4 · p. 8, §6.3 · p. 9
- The results do not show that online interventions reduce data requirements: when methods reach comparable performance, they do so after similar numbers of interactions.The authors suggest PDGD’s lack of repeated passes over prior interactions, unlike counterfactual methods’ numerous epochs, as a possible reason.Source: §6.3 · p. 9
7 RELATED WORK
Prior work has surveyed counterfactual and online LTR, but the cited surveys and comparisons do not systematically compare the two families. This study addresses that gap by comparing both approaches using user interactions.
7 RELATED WORK
- The cited LTR benchmark comparison concerns manually annotated data, whereas this paper compares online and counterfactual methods that use user interactions.The compared methods are not supervised in the traditional sense [16].Source: §7 · p. 9
- Earlier counterfactual LTR overviews discuss theory and deployment but do not provide a large-scale experimental comparison or contrast counterfactual with online methods.The cited works include Joachims and Swaminathan and Ai et al..Source: §7 · p. 9
- The cited online LTR survey lacks a large-scale comparison and does not contrast online with counterfactual methods, while a tutorial compares two online approaches.The survey is, and Oosterhuis compares Dueling Bandit Gradient Descent with PDGD.Source: §7 · p. 9
- The authors identify their work as the first systematic comparison of counterfactual and online LTR approaches and aim to establish when each is appropriate.Source: §7 · p. 9
8 CONCLUSION
The conclusion recommends choosing counterfactual or online LTR according to selection bias, position bias, and interaction noise, while weighing user-experience risks. It also identifies limits in the propensity assumptions and benchmark scope, and calls for safer deployment methods.
8 CONCLUSION
- CLTR performs better without selection bias and with little position bias or interaction noise, whereas OLTR outperforms it when any of these are high.CLTR can also perform worse than the production ranker under high interaction noise, even when it is proven unbiased.Source: §8 · p. 9
- Unlike CLTR, OLTR initially worsens the user experience relative to the production ranker, then quickly improves substantially over it.Practitioners must weigh the initial decline against the prospect of long-term gains.Source: §8 · p. 9
- High interaction noise can degrade CLTR model quality, and deploying that model can worsen the next model further.The authors recommend evaluating models before deployment, for example with offline or counterfactual evaluation [18] [32].Source: §8 · pp. 9–10
- The comparison supplied CLTR with exact propensities, although realistic settings require estimating them, so estimation errors were not assessed.The study also used one metric on one dataset.Source: §8 · p. 10
- Future work could examine methods for evaluating deployment safety and estimating which CLTR or OLTR algorithm would perform best.The authors frame these as ways to improve user experience with little effort from practitioners.Source: §8 · p. 10
Code and data
The paper shares its resources to support reproducibility.
Code and data
- The authors share all resources used in the paper to facilitate reproducibility.Source: Code and data · p. 10