Source-linked AI summary
Recommendations as Treatments: Debiasing Learning and Evaluation
Tobias Schnabel, Adith Swaminathan, Ashudeep Singh, Navin Chandak, Thorsten Joachims
TL;DR
Recommender-system data is selection-biased because users and systems selectively determine which outcomes are observed, complicating evaluation and training. The paper adapts causal-inference propensity methods to derive unbiased estimators and a propensity-scored matrix factorization method, which substantially improves real-world prediction performance while remaining practical and scalable.
Problem
Selection by users and recommendation systems makes recommender-system data Missing Not At Random, creating a challenge for unbiased evaluation and training.
Method
The paper connects recommendation to causal inference, using propensity-weighted estimation and ERM to develop unbiased evaluators and a scalable matrix factorization method.
Results
The estimators can be orders-of-magnitude more accurate than standard estimators, and the matrix factorization method substantially outperforms methods that ignore selection bias and existing MNAR methods.
Takeaways & Limitations
Separating assignment and rating models makes propensity scoring a practical, modular alternative that retains efficiency, predictive performance, and scalability.
Takeaways & Limitations
Propensity estimation assumes observations are independent given P for the stated IPS variability analysis, and assumes assignment is independent of new predictions after observable features are accounted for.
Abstract
from arXiv · showhide
Most data for evaluating and training recommender systems is subject to selection biases, either through self-selection by the users or through the actions of the recommendation system itself. In this paper, we provide a principled approach to handling selection biases, adapting models and estimation techniques from causal inference. The approach leads to unbiased performance estimators despite biased data, and to a matrix factorization method that provides substantially improved prediction performance on real-world data. We theoretically and empirically characterize the robustness of the approach, finding that it is highly practical and scalable.
1. Introduction
The paper treats selection-biased recommendation data as a causal-inference problem and develops unbiased methods for evaluating and training recommender systems. Its propensity-based framework is theoretically grounded, empirically effective, and scalable.
- Recommendation data is typically Missing Not At Random because users and systems selectively expose or rate items.Users tend to rate movies they like, while recommendation systems preferentially display items they expect users to like.
- The paper interprets exposing a user to an item as an intervention analogous to treatment exposure in causal inference.This framing targets estimation of new recommendation policies despite incomplete and biased data from self-selection or system effects.
- It derives propensity-weighted unbiased estimators for multiple performance measures and an ERM framework with generalization error bounds.The estimators cover measures including MSE, MAE, and DCG.
- It derives a conceptually simple, highly scalable matrix factorization method that accounts for selection bias.The method is developed within the propensity-based ERM framework.
- The estimators can be orders-of-magnitude more accurate than standard estimators, while the matrix factorization method substantially outperforms methods that ignore selection bias and existing MNAR methods.These findings come from an extensive empirical evaluation spanning evaluation and learning tasks.
2. Related Work
Prior work on MNAR recommendation data largely uses complex joint-likelihood models, while related approaches include positive-feedback methods and estimators designed to reduce sensitivity to selection bias. Propensity-based weighting draws on causal inference and missing-data methods but is presented here as new to matrix completion.
- Earlier MNAR recommendation methods model missingness and ratings jointly through sophisticated, highly complex likelihood-based approaches.The paper instead separates the assignment and rating models to obtain a modular and scalable approach.
- Positive-feedback recommendation and other MNAR methods address related but distinct settings, including performance measures intended to be less affected by selection bias.Popularity can serve as a propensity proxy in one closely related approach.
- Propensity-based methods are established in causal inference, complete-case missing-data analysis, survey sampling, domain adaptation, and covariate shift.The paper states that their use in matrix completion is new to its knowledge.
3. Unbiased Performance Estimation for Recommendation
Selection bias makes conventional evaluation unreliable because observed ratings depend on user or system choices. The paper reframes recommendation evaluation as causal intervention estimation and uses propensity-based estimators to recover unbiased performance despite incomplete observations.
- Motivation: Selection bias arises when ratings or feedback are observed conditional on users’ or systems’ choices, producing Missing Not At Random data.Users preferentially rate liked movies, while recommendation systems preferentially display ads believed to interest users.
- Task 1: Rating prediction accuracy: The naive estimator averages loss only over observed entries, so it can misrank prediction matrices and is not unbiased under selection bias.In the toy example, it ranks ˆY2 better than ˆY1 despite ˆY1 being better by reasonable performance measures.
- Task 2: Recommendation quality: Recommendation quality requires estimating counterfactual outcomes for items users did not watch, using measures such as CG, DCG, and PREC@k.The target is how users would have benefited from following the recommendations rather than their observed choices.
- Causal formulation: The paper models recommendations as interventions and connects evaluation to estimating policy effects in causal inference with partially observed user-item outcomes.This provides a common framework for rating-accuracy and recommendation-quality estimation.
- Propensity-scored estimators: The IPS estimator is unbiased under any probabilistic assignment mechanism and requires only marginal observation probabilities, even when observations are dependent.The paper assumes non-zero propensities for every user-item pair; propensity estimation is needed in the observational setting.
- Robustness and empirical results: IPS variability increases with non-uniform propensities, while SNIPS can reduce variability for MAE but not DCG.The paper explicitly treats this variability as the cost of unbiasedness and evaluates whether that cost is worthwhile.
4. Propensity-Scored Recommendation Learning
The paper adapts empirical risk minimization to recommendation by weighting observed losses with propensities, yielding unbiased learning under selection bias. This framework supports scalable matrix factorization, generalization guarantees, and extensions beyond standard matrix factorization.
- 4. Propensity-Scored Recommendation Learning: Propensity-scored ERM adapts empirical risk minimization to recommendation using unbiased estimators for losses under biased observations.The framework is stated for training observations, a prediction hypothesis space, and a loss function, with a generalization error bound for finite hypothesis spaces.
- 4.2. Propensity-Scored Matrix Factorization: The resulting matrix factorization objective differs from standard incomplete matrix factorization by weighting each loss term with its propensity.The model includes user, item, and global offsets; existing optimization algorithms can solve the weighted objective efficiently at scale.
- 4.2. Propensity-Scored Matrix Factorization: Conventional incomplete matrix factorization is recovered when all propensities are equal, corresponding to Missing Completely At Random data.Other recommendation methods can in principle be adapted to propensity scoring as well.
5. Propensity Estimation for Observational Data
For observational recommendation data, the paper studies learning with estimated rather than known propensities. It characterizes the resulting bias–variance trade-off and outlines propensity models based on Naive Bayes and logistic regression.
- 5. Propensity Estimation for Observational Data: Effective learning does not require perfectly reconstructed propensities; estimates better than uniform observation probabilities can suffice.The paper characterizes better propensities through the bias they induce and their effect on learning.
- 5. Propensity Estimation for Observational Data: The generalization-error bound for estimated propensities applies to finite prediction hypothesis spaces with positive estimated propensities and independent Bernoulli observations.The bound characterizes the overall impact of estimated propensities on the learning process.
- 5. Propensity Estimation for Observational Data: Estimated propensities create a bias–variance trade-off, so overestimating small propensities may be beneficial when variance reduction outweighs added bias.This trade-off is absent from conventional empirical risk minimization.
- 5. Propensity Estimation for Observational Data: Propensities may depend on observable features, hidden features, and ratings, with independence from new predictions assumed after accounting for observable features.The paper notes that domain-specific propensity-estimation techniques can also be used.
- 5.1. Propensity Estimation Models: Naive Bayes estimates propensities from observed ratings but requires a small Missing Completely At Random sample to estimate the marginal rating distribution.The approach assumes dependencies among covariates and other ratings are negligible.
- 5.1. Propensity Estimation Models: Logistic regression estimates propensities without an MCAR sample by modeling observation as conditionally independent of hidden features and ratings given observable information.The model uses user-item features with per-item and per-user offsets and a sigmoid function.
6. Empirical Evaluation
The evaluation studies propensity-weighted estimators and matrix-factorization learning under controlled and estimated selection bias, including semi-synthetic and real-world settings. Across these settings, the proposed methods improve accuracy and remain robust as bias or propensity-estimation quality varies.
- Evaluation under selection bias: Semi-synthetic experiments compare propensity-weighted IPS and SNIPS estimators with the Naive estimator for MSE, MAE, and DCG.The study varies observation bias using an ML100K-based observation model and evaluates estimators against known performance.
- Evaluation under selection bias: Orders-of-magnitude lower estimation error occurs for IPS and SNIPS than Naive over most bias levels, including α = 0.25.At α = 1, SNIPS equals Naive algebraically; IPS incurs a small variability penalty, while SNIPS is better than IPS for MSE and tied for DCG.
- Learning under selection bias: Propensity-weighted matrix factorization MF-IPS is evaluated against conventional MF-Naive under the same semi-synthetic observation model.The experimental design uses the same factorization model, separate regularization selection, and d = 20 across 30 trials.
- Robustness to propensity estimation: IPS and SNIPS never perform worse than Naive as propensity estimates degrade, and IPS-NB can outperform IPS with known propensities for MSE.The authors partly attribute this effect to estimated propensities providing stratification-like behavior.
- Robustness to propensity estimation: MF-IPS-NB outperforms MF-Naive even with severely degraded propensity estimates, demonstrating robustness in the observational setting.The comparison uses estimated Naive Bayes propensities and includes MF-IPS with known propensities as a reference.
- Performance on real-world data: The authors conclude that discriminative MF-IPS performs robustly and efficiently on real-world data compared with complex generative HL models.They identify further directions including propensity clipping, doubly robust estimation, and improved propensity estimation.
7. Conclusions
The paper proposes propensity scoring as a practical, robust approach for handling selection bias in recommender-system evaluation and training. Its modular discriminative design separates assignment and rating models while retaining efficiency and scalability.
- Conclusions: Propensity scoring provides an effective and robust approach to selection bias in recommender-system evaluation and training.The approach is presented as a discriminative alternative to generative joint-likelihood methods.
- Conclusions: Separating assignment-model estimation from rating-model estimation makes the approach modular and practical.Any conditional-probability estimator can be used for propensities, and existing rating models can in principle be adapted with propensity weighting.
- Conclusions: The discriminative formulation inherits efficiency, predictive performance, fewer modeling assumptions, and no need for latent variables.These advantages are stated relative to the generative methods discussed in the paper.