Synthetic Controls in Action

1. Introduction

• The article distills formal properties of synthetic control estimators into simple principles for empirical practice.The principles address over-fitting, interpretability, validation, trimming, covariates, and model applicability.Source: §1 · pp. 1–2
• The paper demonstrates the practical relevance of its principles through simulation exercises using varied data configurations.These exercises study pre-treatment fit, over-fitting, and the resulting estimator performance.Source: §1 · p. 2
• Close pre-treatment fit is important for estimator performance, but it does not guarantee good performance because over-fitting can still occur.The article examines how over-fitting induces bias and how study design can minimize or avoid it.Source: §1 · p. 2
• The article discusses validation exercises that can assess the validity of synthetic control estimates in empirical applications.It also considers trimming to alleviate over-fitting and interpolation biases, and the role of covariates.Source: §1 · p. 2
• The analysis develops synthetic-control results under a linear factor model and uses an auto-regressive model to illustrate broader applicability.The article is organized around formal properties, performance, validation, trimming, covariates, an auto-regressive illustration, and a conclusion.Source: §1 · p. 2

• Synthetic control estimates the treated unit’s post-intervention counterfactual by comparing its observed outcome with a nonnegative, unit-summing weighted average of untreated donor outcomes.Data-driven weights minimize pre-intervention predictor imbalance, with predictor standardization or emphasis based on variance or predictive power.Source: §2 · pp. 2–3
• Synthetic controls are projections onto the donor pool’s convex hull and are typically sparse, but the treated predictor vector may lie outside that hull in moderately high dimensions.Outside the convex hull, general-position columns yield a unique solution with no more than k nonzero weights; inside it, sparse solutions with no more than k+1 nonzero weights exist, though uniqueness is not guaranteed.Source: §2 · p. 4
• Nonnegative, sparse weights summing to one make contributions and potential contamination or idiosyncratic-shock biases interpretable, while penalized selection can favor nearby donors and reduce interpolation bias.Abadie and L’Hour (2021) provide a procedure producing unique, sparse estimates even when the treated vector lies within the donor convex hull.Source: §2 · p. 4
• Under a linear factor outcome model with zero-mean transitory shocks conditional on observed and unobserved predictors, the synthetic-control bias bound rises with shock scale relative to pre-treatment periods, donor-pool size, and unobserved-factor dimension.The model does not directly restrict treatment-selection processes, and the bound motivates including observed predictors and pre-treatment outcomes among matching variables.Source: §2 · pp. 4–5
• Matching variables should include observed predictors and pre-treatment outcomes because they help control outcome differences, although the model may hold only approximately when the treated vector lies outside the donor convex hull.Source: §2 · p. 5
• Over-fitting arises when volatile pre-treatment outcomes are matched through transitory shocks; it is less likely with small shocks and many pre-treatment periods but more likely with large donor pools.Over-fitting can make fitted unobserved factors differ substantially from the treated unit’s factors, inducing bias.Source: §2 · pp. 6–7
• The seven principles recommend low volatility, an extended pre-intervention period, and a small donor pool of predictor-close untreated units to reduce over-fitting and interpolation biases.A close fit should persist when it reflects shared responses to unobserved factors, whereas distant units can create interpolation bias under nonlinear data-generating processes.Source: §2 · p. 7
• The remaining design principles are interpretability through sparsity, careful inclusion of covariates, and close fit, although deficient fit has exceptions and qualifications (Ferman, 2021; Ferman and Pinto, 2021).Uncontrolled predictor components enter the unobserved factors, making them harder to fit than observed covariates.Source: §2 · p. 7
• Out-of-sample validation is key: predictive power in held-out pre-intervention periods assesses whether the synthetic control predicts the treated unit’s untreated trajectory.The authors study validation techniques derived from this principle alongside the other six principles.Source: §2 · pp. 7–8
• Simulations across data-generating processes investigate these principles, whose first six derive from the Abadie et al. (2010) bias bound despite its strong assumptions.The simulations examine how data features affect estimator performance and whether the principles translate into empirical practice.Source: §2 · p. 8
• Synthetic controls are powerful and transparent design tools for aggregate interventions when applied conscientiously, but mechanical use leaves estimates vulnerable to over-fitting and predictor discrepancies.The analysis stresses that contextual and data requirements should be carefully checked in empirical applications.Source: §2 · p. 8

Baseline simulations

• As σ decreases, pre-treatment fit improves, producing gains in post-intervention estimation precision; with large σ, poor fit translates into large post-treatment estimation errors.Source: §3 · p. 9

Simulation validation

• Across 10000 simulations, estimation accuracy increases with pre-intervention fit, confirming the baseline simulation pattern.The article reports single random simulations in the main text and analogous many-simulation results in the appendix.Source: §3 · pp. 9–10, Figure 3 · p. 11

Stochastic trends

• Non-stationary factors reduce over-fitting: with ρ = 1, the correct unit’s weight remains close to one across σ values.Large factor variation cannot be compensated by transitory shocks, producing minimal over-fitting.Source: §3 · p. 12

Over-fitting

• Good pre-treatment fit is necessary but insufficient because transitory shocks can generate over-fitting without reproducing the treated unit’s systematic component.Researchers should identify settings prone to over-fitting and modify study designs to prevent, detect, or correct its resulting biases.Source: §3 · p. 12
• Figure 6 shows that a small number of pre-intervention periods, T0, can induce over-fitting even when pre-treatment fit is perfect, breaking its correspondence with post-treatment fit.Source: §3 · p. 12

Stochastic trends

• With stochastic trends, heterogeneity helps select controls sharing the treated unit’s trend, yielding large gains over averaging untreated outcomes.Non-stationary factor components help identify controls that reproduce the treated unit’s systematic component, and the method adapts to heterogeneous outcome trends.Source: §3 · pp. 10–11, Figure 4 · p. 13

Over-fitting and interpolation bias

• Perfect pre-treatment fit need not imply accuracy: in one setting, post-treatment estimation error is much larger despite better pre-treatment fit and a zero contribution from the second unit.The results show that over-fitting can arise when the donor pool is large relative to the available pre-treatment information.Source: §3 · pp. 13–14
• Figure 6 illustrates that short pre-treatment periods and many donor units crucially increase synthetic-control bias, even when pre-treatment fit improves.The number of pre-treatment periods and untreated units materially affects estimator bias, with excessive J creating over-fitting risk.Source: §3 · p. 14, Figure 6 · p. 15
• Interpolation bias arises when local linearity fails and treated predictors are reproduced using distant donor units; restricting donors to predictor-similar units can attenuate it.Researchers should recognize small T0, large J, and substantial predictor differences as bias-conducive settings and adapt the design accordingly.Source: §3 · pp. 15–16

Design implications

• Researchers should adapt donor-pool and estimator designs to mitigate bias and use validation techniques to assess potential bias in concrete applications.Additional flexibility may help when treated outcomes are extreme or level fit is unattainable, but it increases over-fitting risk.Source: §3 · pp. 16–17

Modifications of the synthetic control design

• Figure 7 shows pre-treatment fit improving, with pre-RMSE dropping from 0.23 to 0.14, while unrestricted regression perfectly fits pre-treatment data (pre-RMSE 0) and has post-RMSE 0.54.Source: §3 · pp. 16–17
• Nonnegative weights summing to one regularize synthetic controls, whereas relaxing weight restrictions improves flexibility but reduces interpretability and permits unchecked extrapolation beyond observed support.Such extrapolation can produce extreme counterfactuals about which the data contain little information.Source: §3 · p. 18

Validation design

• Backdating treatment creates hold-out periods that can validate synthetic-control predictions before post-treatment data are observed.Weights are estimated using data through T_b, leaving later pre-treatment periods available for validation and study-design decisions.Source: §4 · p. 18

Over-fitting

• Greater flexibility can improve pre-treatment fit while increasing the potential for over-fitting and weakening estimation precision.Source: §4 · p. 19

Validation results

• Large out-of-sample errors occur for σ = 2 and σ = 1, whereas predictions are more reliable for σ = 0.5 and especially σ = 0.25.Source: §4 · p. 19
• Post-treatment prediction quality closely mirrors hold-out prediction error, supporting hold-out performance as a validation signal.Source: §4 · pp. 19–20

Backdating with treatment effects

• With a positive treatment effect, backdating detects an effect at the true treatment time and produces similar magnitudes across validation and treatment panels.Good hold-out fit and similar estimates provide additional credibility to the synthetic-control estimates.Source: §4 · p. 20

Trimming principle

• Trimming donor units by predictor proximity can substantially improve synthetic-control estimator performance.Source: §5 · p. 20

Simulation evidence

• A large donor pool promotes over-fitting and produces large post-treatment estimation errors.The simulation uses J = 199 donor units before trimming.Source: §5 · p. 20
• Retaining the 50, 20, or 10 donor units closest in predictor values is the simulated trimming strategy.Source: §5 · p. 20

Simulation results

• Trimming mechanically increases pre-treatment differences but noticeably improves post-treatment performance.Source: §5 · p. 20

Simulation design

• The simulation incrementally shifts five covariates from unobserved to observed while using J = 1000 and T0 = 4.Source: §6 · pp. 20–21

Covariate inclusion

• Including observed predictive covariates in X_j substantially decreases estimation error as the number of unobserved factors falls.Source: §6 · p. 22
• Adding covariates slightly deteriorates pre-treatment outcome fit because variables beyond outcomes enter X_j.Source: §6 · p. 22

Interpretation

• Observed covariates make estimates more reliable by reducing reliance on pre-treatment outcomes as imperfect proxies.Source: §6 · p. 22

Scope

• The section illustrates synthetic-control applicability under an auto-regressive data-generating process without covariates.Source: §7 · pp. 22–23

Data-generating process

• The auto-regressive design allows coefficients α_jt to vary across units and time, with five groups of 10 units sharing within-group trajectories.Source: §7 · p. 25

Simulation results

• Synthetic-control estimates closely follow the treated unit’s trajectory under the auto-regressive simulation.The result demonstrates applicability outside the linear factor model considered earlier.Source: §7 · p. 25
• The average outcome among untreated units cannot reproduce the treated unit’s trajectory before or after treatment because of group-level heterogeneity.Source: §7 · p. 25

8. Conclusion

• Synthetic controls can provide reliable estimates for a variety of settings, but like other statistical methods may fail in some data-generating processes.Source: §8 · p. 26
• The article describes when over-fitting arises, how to recognize it, and ways to alleviate or eliminate its biases.Source: §8 · p. 26
• The authors present intuitive and effective diagnosis checks for assessing the validity of synthetic control estimates in empirical research.Source: §8 · p. 26
• Synthetic controls can fit treated units using relatively small donor pools, validate out-of-sample, and produce effects at actual treatment timing.Source: §8 · p. 26
• The recommendations are summarized in seven guiding principles for the research design of empirical studies using synthetic controls.Source: §8 · p. 26

Appendix

• Table A.1 summarizes simulation results for all figures, including average pre-RMSE and post-Source: Appendix · p. 29, Table A.1
• Figures A.4–A.12 show 95% bands for estimation error over 10,000 simulations using the same designs as Figures 4–12.Source: Appendix · p. 29
• The appendix examines synthetic-control error under stochastic trends, ρ = 1, over-fitting, and different weight restrictions.Source: Figure A.4 · p. 31, Figure A.5 · p. 32, Figure A.6 · p. 33
• The cited notes specify 95% bands for the simulation design corresponding to each of Figures 4–9, with the same band level stated separately for every figure.Source: Appendix · pp. 31–36
• Additional simulations consider a validation period, treatment effects, trimming, observed covariates, and an auto-regressive process.Source: Figure A.8 · p. 35, Figure A.9 · p. 36, Figure A.10 · p. 37