Source-linked AI summary
Incremental Recommendation via Causal Models
Athanasios Vlontzos, David Gustafsson, Michael O'Riordan, Ciarán M. Gilligan-Lee
TL;DR
Recommendation systems must distinguish streams caused by recommendations from streams users would discover organically. The paper extends a production model with holdback data, resolves mismatched attribution windows using dual-threshold targeting, and finds fewer impressions without a statistically significant reduction in recommended-content consumption.
Problem
Recommendation impressions are scarce, and standard stream-probability optimization favors users who would stream organically rather than measuring incremental impact.
Method
The paper extends an existing production recommendation model into a causal architecture using already-collected holdback data and a dual-threshold targeting policy.
Results
7% fewer recommendation impressions produced no statistically significant reduction in recommended content consumption or listener satisfaction guardrails in a production-scale test.
Takeaways & Limitations
Joint training with holdback data improves treated-head calibration, which the authors argue is evidence of more generalisable causal-model representations.
Takeaways & Limitations
Naive treatment-effect subtraction is not causally interpretable because treated and holdback outcomes use different attribution windows and definitions.
Abstract
from arXiv · showhide
Recommendation impressions are a finite resource, hence delivering a recommendation to a user who would discover the content organically yields no incremental value and displaces other recommendations that could. We address this by extending an existing production recommendation model to a causal architecture using holdback data that is already collected as part of routine experimentation infrastructure, requiring no new data collection. A central challenge is that attribution windows differ between treated and holdback observations: treated users are attributed a stream within a short direct-response window, while holdback users are attributed organic streams over a multi-day window. This mismatch makes naive treatment-effect subtraction invalid. We resolve this with a dual-threshold targeting policy that delivers a recommendation only when the probability of a treated stream is high and the probability of organic stream is low. In a production-scale A/B test on millions of Spotify users, this policy reduces recommendation impressions by 7% with no statistically significant reduction in overall recommended content consumption. We further show that joint training with holdback data improves calibration of the treated head relative to the production baseline, and argue this can be taken as evidence that causal models learn more generalisable representations than models trained on observational data alone.
1 Introduction
The paper reframes recommendation as an incremental-impact problem: impressions should target users whose behavior changes because of the recommendation. It uses existing holdback data to build and validate a causal targeting system without new data collection.
- Standard recommenders maximize the probability of a stream but do not ask whether the user would have streamed organically.
- Always-takers are easy to predict yet receive non-incremental recommendations, displacing users for whom recommendations could make a genuine difference.
- Holdback experiments provide direct samples of organic counterfactual behavior, but are rarely used to build causal recommendation models.
- The proposed approach converts an existing production model into a causal incremental model using holdback data and no new data-collection infrastructure.
- 7% fewer recommendation impressions produced no statistically significant reduction in recommended content consumption in a production-scale Spotify evaluation.
- Joint training with holdback data improves treated-head calibration relative to the production baseline, which the authors argue indicates more generalisable representations.
2 Problem Setting
The problem setting models recommendation delivery as a treatment decision and defines incremental impact through potential outcomes. Randomised holdback observations provide unbiased information about organic behavior, while naive outcome subtraction is invalid when outcome definitions differ.
- For each user–content pair, the system makes a binary decision: show the recommendation (T = 1) or withhold it (T = 0).
- The CATE measures how much showing a recommendation increases the probability of a stream for a given user–content pair.
- A user with τ(x) ≈ 0 is a sure thing who streams with or without the recommendation, making the impression non-incremental.
- Randomised holdback assignment makes withheld observations unbiased samples from the organic outcome distribution Y(0).
- Naive CATE subtraction, p̂1(x) − p̂0(x), is invalid because the treated and holdback outcomes use structurally different attribution definitions.
3 The Attribution Mismatch
The treated and holdback outcomes use different attribution windows, so their probabilities are not directly comparable. This mismatch invalidates naive subtraction, although each score remains useful for targeting decisions.
- Attribution windows: Treated outcomes use a short direct-response window, whereas holdback outcomes use a two-day organic window.The treated window captures streams plausibly caused by the impression; the holdback window captures slower organic discovery.
- Why subtraction fails: Different outcome definitions make ˆp1(x) − ˆp0(x) invalid as a CATE estimator.The subtraction compares probabilities measured on different scales and therefore has no clean causal interpretation.
- Distributional mismatch: A roughly 30% discrepancy between backend servings and client-side impressions creates an additional distributional mismatch between treated and holdback examples.Latency and client-side rendering differences mean the two datasets are not drawn from identical distributions over x.
- Decision signals: High ˆp1(x) signals likely recommendation-driven streaming, while high ˆp0(x) signals likely organic streaming.The two scores therefore provide separate decision-relevant signals rather than a directly subtractable treatment effect.
4 Causal Recommendation Model
The causal model extends the production shared-trunk recommender with a holdback head and trains the two outcome heads on their respective treatment conditions. A dual-threshold policy then serves recommendations only when treated-stream probability is high and organic-stream probability is low.
- 4.1 Deep Twin Network Extension: The production baseline uses a shared-trunk multi-task network trained exclusively on treated examples to predict ˆp1(x).Its task-specific heads predict engagement outcomes, while the primary head drives targeting decisions.
- 4.1 Deep Twin Network Extension: The causal extension adds a holdback head that predicts ˆp0(x) from holdback observations while retaining the shared trunk.This provides a separate model output for organic-stream probability.
- 4.1 Deep Twin Network Extension: Treated examples update only the treated head, holdback examples update only the holdback head, and both update the shared trunk.This partitioned gradient flow jointly shapes the representation without conflating the different attribution windows.
- 4.2 Dual-Threshold Targeting Policy: The policy serves a recommendation only when ˆp1(x) ≥ θ1 and ˆp0(x) ≤ θ0.The first threshold selects likely recommendation-driven streams; the second excludes likely always-takers.
- 4.2 Dual-Threshold Targeting Policy: Increasing θ0 withholds more impressions from users with higher organic-stream probability, trading impression volume against potentially missed incremental conversions.Thresholds are tuned offline to meet impression-reduction targets while satisfying non-inferiority constraints.
5 Related Work
Related work spans uplift and CATE estimation, causal recommendation, and impression efficiency. This paper emphasizes production-scale incremental recommendation using existing holdback infrastructure and mismatched outcome scales.
- Uplift modelling and CATE estimation: Uplift modelling and CATE methods estimate individual-level treatment effects, including neural approaches such as TARNet, CFR, and DragonNet.The paper situates its setting in production systems with tens of millions of users and differently scaled treated and holdback outcomes.
- Uplift modelling and CATE estimation: Theoretical work highlights the difficulty of validating causal models against experimental data in production deployments.This motivates careful experimental design for causal recommendation systems.
- Causal recommendation systems: Causal recommendation research includes propensity weighting, counterfactual offline evaluation, treatment-aware modelling, and causal representation learning.The paper differs by explicitly targeting incrementality rather than primarily debiasing click or conversion prediction.
- Impression efficiency: The paper demonstrates that existing holdback infrastructure can support production-scale incremental recommendation with impression savings without degrading recommendation impact.This connects the always-taker problem from direct marketing to recommendation systems.
6 Experiments
A production-scale A/B test compared causal recommendation targeting with treated-stream targeting alone on Spotify’s Home surface. The dual-threshold policy reduced impressions while preserving recommended-content consumption and satisfaction guardrails.
- Experiment setup: Millions of Spotify users participated in a live A/B test of the causal targeting approach on the Home recommendation surface.The experiment operated at production-serving latency and throughput constraints.
- Experiment setup: The three arms separated the effects of causal joint training from the dual-threshold policy.Treatment-Model used the causal model with treated-stream filtering; Treatment-Causal added the dual-threshold policy, with their comparison isolating policy effects.
- Metrics: The test tracked recommendation impressions per user as the efficiency metric and recommended-content consumption minutes as the impact metric.Non-inferiority was required for consumption, and listener satisfaction served as a guardrail.
- Results: 7% fewer recommendation impressions resulted from the dual-threshold policy relative to treated-stream targeting alone.The comparison was Treatment-Causal versus Treatment-Model.
- Results: Recommended-content consumption and listener satisfaction showed no statistically significant change under the dual-threshold policy.The policy therefore removed impressions without a statistically significant reduction in the reported impact metric or satisfaction guardrails.
- Interpretation: Approximately 93% of production-model impressions were estimated to be incremental, leaving 7% identified as non-incremental.The paper attributes the removed impressions to users who would discover the content organically.
7 Calibration and Generalisation
Joint training with holdback data improves treated-head calibration relative to the production baseline, especially at higher predicted probabilities. The paper links this improvement to richer representations of affinity and recommendation sensitivity, while noting smaller holdback samples cause minor deviations.
- Calibration: Calibration matters because inaccurate predicted probabilities undermine the interpretation and tuning of the policy thresholds.The thresholds are θ1 and θ0.
- Calibration: The causal treated head tracks the calibration diagonal more closely than the production baseline, with the largest improvement at higher predicted probabilities.Figure 4 compares the causal treated head with the production baseline.
- Calibration: The holdback head is well-calibrated, with minor tail deviations attributed to the smaller holdback sample size.The holdback sample is smaller than the treated sample.
- Generalisation: Treated-only training conflates user-content affinity with the effect of showing a recommendation, tending to over-predict for high-affinity users.The production baseline is trained exclusively on treated data.
- Generalisation: Joint holdback training makes the shared trunk represent both affinity and recommendation sensitivity, giving the treated head a more complete picture of user-content affinity.The holdback head must predict organic stream probability, shaping the shared representation across treatment conditions.
- Generalisation: The paper presents improved calibration as an empirical instance of the broader claim that causal models trained across treatment conditions learn representations more robust to distribution shift.This is framed as a hypothesis and broader principle rather than a directly established universal result.
8 Conclusion
The paper transforms a production recommendation model with holdback infrastructure into a causal incremental targeting system. At production scale, it reduces impressions while preserving recommended-content consumption and satisfaction, and improves treated-head calibration.
- Conclusion: The production recommendation model is transformed into a causal incremental targeting system using existing holdback infrastructure.The transformation requires no new data collection infrastructure.
- Conclusion: 7% fewer recommendation impressions were observed with no statistically significant reduction in recommended-content consumption or user satisfaction.This result comes from a production-scale A/B test on millions of Spotify users.
- Conclusion: Joint training with holdback data improved model calibration, which the paper argues indicates more generalisable causal representations.The argument concerns the causal model relative to the production baseline.