PaperlayerView exact revision on arXiv

Source-linked AI summary

Estimating the Causal Impact of Recommendation Systems from Observational Data

Amit Sharma, Jake M. Hofman, Duncan J. Watts

arXiv:1510.05569v1cs.SIstat.AP

TL;DR

Recommendation clicks may overstate causal impact because users might have found recommended products through other means, and randomized experiments can be costly or inconvenient. The paper estimates effects from observational logs by using focal-product traffic shocks when recommended products do not experience corresponding demand changes. In Amazon data, only a quarter of recommendation click-throughs on shocked products were considered causal, with generalization bounded by the method’s scope and assumptions.

  • Problem

    Counting recommendation clicks does not establish how much activity recommendations cause, because users may have visited those products through other means; randomized experiments can be costly and may inconvenience users.

    Source: §1 · p. 1

  • Method

    The method uses focal-product traffic shocks as instruments when direct traffic to recommended products remains constant, identifying effects from observational logs.

    Source: §1 · p. 3, §4.1 · p. 9

  • Results

    Only a quarter of recommendation click-throughs on shocked Amazon products can be considered causal.

    Source: §6 · p. 17

  • Takeaways & Limitations

    The method requires separate counts of recommendation-driven traffic and total page visits over time, which the authors describe as inexpensive compared with A/B tests.

    Source: §6 · p. 16

    The estimates are local to users responding to product shocks and may not represent Amazon-wide effects; because shocks affect one focal product at a time, the authors expect their causal click-through rates to overestimate site-wide impact.

    Source: §5.3 · p. 15, §6 · p. 17

Abstract

from arXiv · show

Recommendation systems are an increasingly prominent part of the web, accounting for up to a third of all traffic on several of the world's most popular sites. Nevertheless, little is known about how much activity such systems actually cause over and above activity that would have occurred via other means (e.g., search) if recommendations were absent. Although the ideal way to estimate the causal impact of recommendations is via randomized experiments, such experiments are costly and may inconvenience users. In this paper, therefore, we present a method for estimating causal effects from purely observational data. Specifically, we show that causal identification through an instrumental variable is possible when a product experiences an instantaneous shock in direct traffic and the products recommended next to it do not. We then apply our method to browsing logs containing anonymized activity for 2.1 million users on Amazon.com over a 9 month period and analyze over 4,000 unique products that experience such shocks. We find that although recommendation click-throughs do account for a large fraction of traffic among these products, at least 75% of this activity would likely occur in the absence of recommendations. We conclude with a discussion about the assumptions under which the method is appropriate and caveats around extrapolating results to other products, sites, or settings.

1. INTRODUCTION

Recommendation clicks can overstate causal impact because users may have visited recommended products anyway through correlated demand or other routes. The paper proposes identifying causal effects from observational logs by using shocks to focal-product traffic when demand for recommended products remains constant.

1. INTRODUCTION

  • Correlated demand makes recommendation-click counts overstate causal views, since interested users may find recommended products through search even without recommendations.Randomized experiments can estimate causal impact, but are costly and may inconvenience users.Source: §1 · pp. 1–2
  • Earlier natural experiments using focal-product traffic shocks can still confound recommendation clicks when the shock also raises demand for recommended products.The paper illustrates this with a book promoted alongside other works by the same author.Source: §1 · p. 2
  • Causal identification is possible when a focal product receives an exogenous traffic shock while demand for its recommended products stays constant.The shock serves as an instrument; the estimated effect is the change in recommendation click-throughs divided by the exogenous traffic change.Source: §1 · p. 2
  • At least 75% of recommendation click-through activity on shocked products would likely occur without recommendations, based on analysis of more than 4,000 qualifying shocks.The study uses Amazon browsing data from 2.1 million users over nine months.Source: §1 · p. 3
  • The paper formalizes the assumptions, constructs instrumental variables from logs, and applies the method to Amazon data, while discussing limits to generalization.The dataset contains 23 million visits by 2.1 million Bing Toolbar users to 1.38 million products over nine months.Source: §1 · p. 3

2. RELATED WORK

Prior work evaluates recommenders across many dimensions, but estimating their causal effect on traffic is difficult. This paper distinguishes its observational strategy by measuring pageview demand directly and selecting shocks that leave recommended-product traffic constant.

2. RELATED WORK

  • Recommendation research evaluates accuracy, diversity, utility, novelty, and serendipity; this paper focuses on whether recommendations increase website traffic by exposing users to items they otherwise would not see.The paper notes that this narrower focus overlooks other potentially important recommender functions.Source: §2 · p. 3
  • Randomized studies found recommendation-related increases, including 5–9% more pageviews per user in one experiment, while some earlier studies lacked a no-recommendation control.The cited studies include an online-grocer usage study and a mobile-game experiment.Source: §2 · p. 3
  • Observational studies used covariates, complementary products, or exogenous shocks, but estimates could remain confounded by unobserved demand.Complementary-product methods rely on an assumption that is critical but difficult to verify: that comparison products experience the same demand.Source: §2 · p. 4
  • Unlike prior approaches, this study uses browser-log pageviews as a direct demand measure and selects shocks where recommended products’ direct traffic is constant.It also identifies shocks directly from logs rather than first positing plausible events and then checking their impact.Source: §2 · p. 4

3. DATA

The study reconstructs Amazon product visits and recommendation links from anonymized Bing Toolbar browsing logs, then filters users and products for analysis. Its descriptive data show recommendation traffic varies over time and by category, while the causal analysis focuses on a narrower type of outgoing recommendation click.

3. DATA

  • Referral codes in Amazon URLs let the logs distinguish direct or search visits from recommendation visits and reconstruct links between recommended products.The logs record URLs, timestamps, and anonymized user identifiers; encrypted HTTPS pageviews cannot be reliably identified.Source: §3 · pp. 4–5
  • 23.4 million visits by 2.1 million users to 1.38 million products remained after filtering, covering September 2013 through May 2014.Filters removed likely bots, sellers, merchants, and products with unreliable or sparse product data.Source: §3 · pp. 5–6
  • About 30% of page traffic came through recommendations overall, dipping to about 25% during the holiday season as total visits rose.Figure 2 depicts overall visits and the recommendation fraction from September 2013 to May 2014.Source: §3 · p. 6, Figure 2 · p. 6
  • Recommendation traffic ranged from just under 30% for books and ebooks to over half for shoes and apparel.Figure 3 breaks visits and recommendation fractions down by product category; the passage notes these differences may reflect discovery or correlated demand.Source: §3 · p. 6, Figure 3 · p. 7
  • The causal analysis uses outbound clicks from “Customers who bought this also bought” recommendations, not incoming recommendation traffic of all types.This recommendation type captures most outbound product-page clicks and is consistently defined across categories, so the resulting naive estimates are somewhat lower than the descriptive figures.Source: §3 · p. 7

4. METHODS

The method uses sudden shocks to focal-product traffic, paired with stable direct traffic to recommended products, to identify causal recommendation clicks from observational logs. It operationalizes those conditions with shock filters and selects β = 0.7 as a compromise between identification and estimation.

4.1. Identification Strategy

  • The model separates causal clicks from convenience clicks: users may click a recommendation to reach a product they would otherwise have found.Convenience views represent the potential direct visits that could replace recommendation clicks if the recommendation were removed.Source: §4.1 · p. 9
  • A focal-product shock can identify causal clicks when recommended products' direct traffic stays constant, isolating the shock's effect from correlated demand.The shock serves as an instrument for exposure to the focal product; the estimate is a local average treatment effect for users who respond to shocks, not the population-wide ATE.Source: §4.1 · pp. 7–9

4.2. Shocks

  • 4,774 shocks across 4,126 distinct products met the criteria, and most focal products had no recommendation click-throughs on shock day.The shock-day figure shows the distributions of focal-product visits and recommendation click-throughs.Source: §4.2 · p. 10, Figure 5 · p. 11
  • A stricter 10× shock threshold produced fewer shocks but did not qualitatively alter the results, supporting robustness to the selection cutoff.The authors report the more expansive criteria as their primary specification.Source: §4.2 · p. 10
  • At β = 0.7, 90% of shocks (4,314) remain; stricter constancy requirements sharply reduce the sample and can bias estimated causal rates downward.The authors choose an intermediate β to balance estimation volume against the demand constancy needed for identification.Source: §4.2 · p. 11, Figure 6 · p. 12
  • Accepted shocks show large focal-product increases with only small changes in recommended-product visits, unlike rejected shocks where both rise together.Manual inspection found these accepted and rejected patterns to be typical.Source: §4.2 · pp. 11–12, Figure 7 · p. 12

5. RESULTS

Using shocks to estimate recommendation effects, the paper finds that only about a quarter of observed recommendation traffic is causal. The authors also test robustness and caution that shock-responsive products and users may not represent Amazon-wide activity.

5. RESULTS

  • Alternative shock heuristics and β choices yield qualitatively similar results, provided β is high enough to reduce correlation and low enough to retain events.Alternatives included using median rather than mean preceding traffic and allowing shocks over more than one day.Source: §5 · p. 12

5.1. Estimating the causal click-through rate

  • The estimated causal click-through rate is roughly 4% at β = 0 and closer to 3% at β = 0.7.Increasing β imposes a more stringent control on exogenous demand for recommended products.Source: §5.1 · p. 13, Figure 8 · p. 13

5.2. Estimating the fraction of causal clicks

  • At β = 0.7, about 25% of observed recommendation traffic is estimated to be causal; the remainder is convenience traffic that would have occurred anyway.The estimated causal fraction decreases as β increases.Source: Figure 8 · p. 13, §5.2 · p. 14
  • Causal click-through rates are 5% or lower across product categories, versus naive recommendation conversion rates above 15% for ebooks and toys.Figure 9 contrasts the category-level naive estimates with estimates for shocked products.Source: §5.2 · p. 14, Figure 9 · p. 14

5.3. Generalization issues

  • The estimates are local average treatment effects for users who respond to product shocks and may not represent Amazon-wide average effects.Shock products and participating users may be unrepresentative of Amazon traffic.Source: §5.3 · p. 15

5.3.1. Price discounts on focal products.

  • For ebooks, deal-site referrals account for about 3% of visits on shock days, but excluding them changes results little; no noticeable price changes appeared among 30 inspected products.The authors could not retrieve historical prices through Amazon’s API and manually inspected 30 focal products using a third-party service.Source: §5.3.1 · p. 15

5.3.2. Holiday effects.

  • Although shocks cluster in the holiday period, excluding shocks from November 15 through December 31 produces no significant difference in results.Shocks also occur throughout the nine-month study period.Source: §5.3.2 · p. 15

5.3.3. Distribution of user and product attributes.

  • Shocks are concentrated in different product groups than regular activity, and shock visitors include a higher proportion of high-affinity users than typical traffic.Shocked products are also more popular on average, which may complicate generalization to the full catalog.Source: §5.3.3 · p. 16

6. DISCUSSION

Using direct traffic as a proxy for product demand, the method estimates causal recommendation activity without fitting statistical models. Applied to shocked Amazon products, it finds only a quarter of recommendation click-throughs causal, while the authors caution against broad extrapolation.

6. DISCUSSION

  • The method uses direct traffic as a proxy for product demand and requires only separate counts of recommendation-driven traffic and total page visits over time.The authors say it avoids fitting statistical models or constructing comparable product sets and can also assess contextual advertisements.Source: §6 · p. 16
  • Only a quarter of recommendation click-throughs on shocked Amazon products can be considered causal.Source: §6 · p. 17
  • The study’s estimate may not generalize broadly because it covers one non-personalized recommendation type and a nonrandom set of shocked products.Shocked products skewed toward popular products and certain categories, while some visitors had unusually high interest; raising β also restricted the sample and lowered inferred click-through rates. Post-stratifying estimates would require a larger dataset.Source: §6 · p. 17
  • Because the method estimates effects for one focal product at a time, the authors regard its roughly 8% estimate of traffic caused by recommendations as an upper bound on the recommender’s total site-wide effect.Source: §6 · p. 17
Loading 1510.05569v1…