Source-linked AI summary
Inferring causal impact using Bayesian structural time-series models
Kay H. Brodersen, Fabian Gallusser, Jim Koehler, Nicolas Remy, Steven L. Scott
TL;DR
The paper addresses causal impact estimation for discrete market interventions when the no-treatment response is unobserved. It proposes a Bayesian state-space synthetic-control model and demonstrates a 85,900-click, 21% cumulative lift estimate for an advertising campaign, with uncertainty intervals and temporal impact summaries.
Problem
The paper targets causal impact estimation for discrete market interventions using observed outcomes and an unobserved no-treatment response.
Method
It uses a Bayesian state-space model to construct counterfactual market activity from structural components and control information.
Results
85,900 clicks, or 21%, was the estimated cumulative lift, with a [12%,30%] interval; the estimate was close to the randomized-design baseline of 84,700 clicks.
Takeaways & Limitations
The approach provides pointwise and cumulative impact summaries for evaluating designed market interventions and informing return-on-investment assessment.
Takeaways & Limitations
Inference becomes more uncertain as the post-intervention forecasting period increases, and cumulative impact is uninterpretable for stock outcomes.
Abstract
from arXiv · showhide
An important problem in econometrics and marketing is to infer the causal impact that a designed market intervention has exerted on an outcome metric over time. This paper proposes to infer causal impact on the basis of a diffusion-regression state-space model that predicts the counterfactual market response in a synthetic control that would have occurred had no intervention taken place. In contrast to classical difference-in-differences schemes, state-space models make it possible to (i) infer the temporal evolution of attributable impact, (ii) incorporate empirical priors on the parameters in a fully Bayesian treatment, and (iii) flexibly accommodate multiple sources of variation, including local trends, seasonality and the time-varying influence of contemporaneous covariates. Using a Markov chain Monte Carlo algorithm for posterior inference, we illustrate the statistical properties of our approach on simulated data. We then demonstrate its practical utility by estimating the causal effect of an online advertising campaign on search-related site visits. We discuss the strengths and limitations of state-space models in enabling causal attribution in those settings where a randomised experiment is unavailable. The CausalImpact R package provides an implementation of our approach.
2.1. Components of state.
The model assembles additive state components for local trends, seasonality, and contemporaneous controls, with static or time-varying regression coefficients. These components support synthetic-control counterfactual prediction while allowing flexible temporal dynamics.
- Local linear trend: The local linear trend represents a time-varying level and slope that evolve through Gaussian innovations.The slope δt is the expected increase in the level µ between times t and t + 1.
- Local linear trend: Local linear trends adapt quickly to local variation, but longer-term predictions can have implausibly wide uncertainty intervals.This flexibility is desirable for short-term prediction but may be unsuitable for longer horizons.
- Seasonality: Seasonal components represent S seasonal effects whose joint contribution is constrained to sum to zero across seasons.The framework also supports multiple seasonal components with different periods, such as day-of-week and annual cycles.
- Regression components: Regression components use untreated control series to capture shared variance and construct synthetic-control counterfactuals.Static coefficients can be selected with spike-and-slab priors, while dynamic coefficients allow control relationships to change over time; the present model assumes contemporaneous covariates unless a known lag is introduced explicitly.
- Model choice: Structural time-series models flexibly combine trend, seasonality, and static or dynamic regression according to the observed time series and modeling assumptions.Dynamic regression can be useful when the relationship between treated metrics and controls changes over time.
- State-model assembly: The model combines independently assembled state components, each contributing additively to the observed response.Figure 2 illustrates a local linear trend paired with static regression.
2.2. Prior distributions and prior elicitation.
The Bayesian model places priors on parameters, initial states, variances, and regression inclusion, with elicitation schemes that encode expected effect sizes, variance, and model size. These priors can remain weakly informative while incorporating domain knowledge and controlling covariate selection.
- Bayesian specification: Bayesian inference specifies a prior on model parameters and a distribution for initial states before sampling their posterior with MCMC.The parameter vector θ and full state sequence α are inferred jointly from the observed data.
- Variance priors: Variance priors encode a prior sum of squares and prior sample-size weight, so s/ν represents a prior estimate of σ².The Gamma distribution used for such variances has expectation a/b.
- Variance priors: Default priors express weak prior beliefs that incremental state-process errors are small, with scale calibrated using the target series variance.The paper notes that scaling by the sample variance is a minor Bayesian-paradigm violation used to choose a reasonable prior scale.
- Regression priors: Spike-and-slab priors combine point mass at zero with a weakly informative distribution for nonzero coefficients, allowing the model to select controls.The spike can be parameterized with independent Bernoulli inclusion indicators.
- Regression priors: Regressor inclusion probabilities can be elicited from an expected model size M, using πj = M/J when equal inclusion probabilities are appropriate.Specific variables can instead be forced into or out of the model by setting their inclusion probabilities to 1 or 0.
- Regression priors: The slab prior uses conjugate normal-inverse Gamma structure, with parameters encoding prior expectations for coefficients and explained variance.A Zellner g-prior interprets g as observations worth of information, while diagonal averaging preserves propriety when XTX is not positive definite.
2.3. Inference.
Inference alternates posterior simulation of states and parameters, then propagates those draws into a joint posterior predictive distribution for the unobserved counterfactual. The resulting samples support uncertainty-aware estimates of pointwise and cumulative causal impact.
- Posterior simulation: Posterior inference first samples model parameters and states conditional on pre-intervention observations.A Gibbs sampler alternates state simulation with parameter simulation.
- Impact summaries: Posterior predictive samples are reused to compute the posterior distribution of incremental impact and other derived summaries.The implementation uses C++ with an R interface and took less than 30 seconds for a typical m = 500, J = 10, 10,000-iteration dataset.
- Posterior simulation: The state draw uses direct simulation from the conditional state posterior based on Gaussian data augmentation, the Kalman filter, and a fast mean smoother.The algorithm is linear in the total number of time points m and quadratic in state dimension d.
- Parameter simulation: Given sampled states, most variance parameters retain Gamma posteriors through conjugacy, while static regression coefficients require a separate sampling procedure.The static-coefficient procedure integrates out β and 1/σ²ε when evaluating the relevant posterior.
- Counterfactual prediction: Posterior predictive simulation generates the counterfactual response for the post-intervention period, where it would have been observed without treatment.The predictive density is conditioned on pre-intervention treatment data and control-market activity before and during the intervention.
- Counterfactual prediction: The predictive distribution is joint across counterfactual time points, preserving serial dependence when summarizing cumulative intervention effects.This avoids treating counterfactual observations as unrelated pointwise predictions.
2.4. Evaluating impact.
The paper evaluates Bayesian counterfactual impact estimates through simulations and an online advertising application. Results show useful temporal attribution and close agreement with randomized and observational-control analyses, while accuracy declines with longer forecasts or structural change.
- Posterior impact summaries: Posterior inference supports cumulative impact estimates for flow outcomes and running averages for both flow and stock outcomes.Cumulative causal increments are interpretable for quantities measured over time, whereas running averages remain interpretable for stock quantities.
- Simulation results: A 1% market lift was missed in about 90% of simulations, whereas a 25% lift was correctly detected in most cases.These empirical detection rates characterize sensitivity for positive effects in the simulated setting.
- Simulation results: Posterior central 95% intervals approximately maintained their intended coverage across campaign durations.The simulations assessed whether interval coverage remained near 95% as campaign length varied.
- Simulation results: Impact estimates became less accurate as the counterfactual forecasting period increased and became considerably less reliable after a model-structure change.Tripling the random-walk standard deviation governing a regression coefficient caused observed data to diverge more quickly and reduced estimation reliability.
- Empirical application: The advertising campaign generated 88,400 additional clicks, a 22% increase with a central 95% credible interval of [13%,30%].The estimated pointwise lift peaked after about three weeks and faded roughly one week after the campaign ended.
- Empirical application: The observational-covariate analysis estimated 85,900 additional clicks, or 21%, with a [12%,30%] interval, closely matching the randomized-control analysis.The randomized analysis estimated 84,700 clicks, and the observational estimate differed from that baseline by about 1%.