Source-linked AI summary
Beyond the Illusion of Power: Calibrating Quasi-Experiments in Observational IS
Spandan Ghose Chowdhury
TL;DR
IS power calculations for DiD and IV often rely on i.i.d. assumptions poorly aligned with observational panels. Using a unified Monte Carlo benchmark, the paper finds the classic DiD anchor’s achieved power materially below its i.i.d. design-based estimate.
Problem
Power calculations based on i.i.d. settings are poorly aligned with IS panels that have serial correlation, attrition, staggered treatment timing, and heterogeneous effects.
Source: §1 · p. 1Method
The paper builds a unified Monte Carlo benchmark for DiD and IV under an IS-realistic data-generating process.
Source: §1 · p. 1Results
0.067 achieved power for TWFE versus 0.291 i.i.d. design-based power in the classic DiD panel anchor.
Source: §4.6 · p. 7Takeaways & Limitations
IV identification requires both instrument relevance and exclusion, which cannot be evaluated through first-stage diagnostics alone.
Source: §5 · p. 7The DGP omits multi-way error dependence, unit-specific trends, and seasonality, making the isolated irreducible gap a conservative lower bound for messier panels.
Source: §6.2 · p. 9
Abstract
from arXiv · showhide
Information systems (IS) researchers increasingly use quasi-experimental methods such as difference-in-differences (DiD) and instrumental variables (IV) to recover causal effects from observational panel data. Power calculations that justify these designs assume i.i.d. errors, but the deeper problem is what even a cluster-robust calculator cannot see. We report a Monte Carlo study over 9837 parameter conditions (approx 9.8 million datasets) and decompose the planned-versus-achieved power gap. The serial-correlation component is recoverable by an AR(1)-aware calculator when rho is known, and partially when rho must be estimated from short pre-periods, but panel attrition, staggered-adoption bias, and parallel-trends pretesting are captured by no closed-form formula; exogenous attrition alone costs approx 8 to 11 percentage points at the few-hundred-to-thousand sample sizes IS studies use. Treatment-correlated, outcome-dependent attrition instead induces bias, not just power loss. For IV, holding first-stage F fixed, larger N neither raises power nor curbs exclusion bias, though with a fixed instrument more data does sharpen the first stage, so identification rests on instrument strength, not sample size.
1. Introduction
The paper asks which parts of the planned-versus-achieved power gap remain after competent cluster-robust, AR(1)-aware calculations, and jointly quantifies established frictions in IS-typical panels. It presents a unified DiD and IV benchmark and translates the results into conditional design guidance.
1. Introduction
- The central question is how much of the power gap remains after a cluster-robust, AR(1)-aware calculator, and which frictions no closed-form formula captures.The study focuses on attrition, staggered timing, pretrend-test conditioning, and IV identification failures.Source: §1 · p. 1
- The study quantifies the joint magnitude of established econometric mechanisms in IS-typical panels using a unified Monte Carlo benchmark for DiD and IV.The mechanisms include serial-correlation distortion, pretest selection, staggered-timing bias, and the weak-versus-exclusion IV trade-off.Source: §1 · p. 1
- The benchmark covers 9,837 parameter conditions, calibrates effect sizes to documented IS magnitudes, validates against canonical open datasets, and offers a companion lookup tool.The tool matches categorical inputs exactly, snaps numeric inputs to the nearest simulated grid value, and reports the matched design.Source: §1 · p. 2
- Weak-instrument bias attenuates with N, whereas exclusion bias is not cured by sample size, so IV identification rests on instrument strength rather than N.The paper frames its design guidance as conditional rather than a universal checklist.Source: §1 · p. 2
2. Background
The background describes why common observational-panel designs pose inferential challenges: staggered adoption and pretrend conditioning complicate DiD, while IV relevance and exclusion are distinct concerns. The paper situates its benchmark among existing methods and treats serial dependence, attrition, and clustering as design inputs.
2. Background
- With staggered timing and heterogeneous effects, TWFE combines multiple 2x2 comparisons, including already-treated versus not-yet-treated contrasts that can receive negative weights and induce bias.The paper benchmarks TWFE against Sun and Abraham and Callaway and Sant’Anna estimators that re-center identification at the group-time level.Source: §2 · p. 2
- Conditioning interpretation on jointly insignificant pre-treatment coefficients can induce selection bias, and conditional error rates can exceed unconditional ones, especially with short, noisy pre-periods.The passage attributes this pretest-selection concern to Roth (2022).Source: §2 · p. 2
- IV requires both relevance and exclusion; geographic, temporal, and peer-exposure instruments may face contextual spillovers, making these distinct empirical concerns.The passage notes that first-stage diagnostics often summarize relevance, while exclusion requires Z to affect Y only through X.Source: §2 · p. 2
- Serial dependence, attrition, and clustering are treated as first-order design inputs because they reduce effective information relative to nominal row counts.The passage connects this reduction to serially dependent, attrition-thinned panels.Source: §2 · p. 2
- The paper does not implement imputation-based, two-stage DiD, matrix-completion, synthetic-control, or causal-ML methods, leaving them for follow-up comparison.Its benchmark jointly analyzes DiD and IV on a panel grid modeling serial correlation and attrition and quantifies estimator performance and inference calibration.Source: §2 · p. 2
3. Simulation Design
The simulation design combines an IS-calibrated panel data-generating process with DiD and IV estimators, multiple performance metrics, and implementation checks. Its grid covers 9,837 parameter conditions and approximately 9.8 million simulated datasets.
3.1. Data-Generating Process
- The experiment spans 6,912 DiD conditions, a 45-cell pretrend study, and 2,880 IV cells, varying panel design, persistence, adoption, heterogeneity, attrition, clustering, and IV inputs.DiD effect-size anchors draw on a structured, though not systematic, review and domain judgment; the IV grid fixes target first-stage F rather than the first-stage coefficient.Source: §3.1 · pp. 2–3, §3.3 · p. 3, §3.2 · p. 3
3.2. Estimators
- The DiD estimators are TWFE, Sun and Abraham, and Callaway and Sant’Anna; the IV analyses use just-identified and three-instrument 2SLS.The pretrend diagnostic is a joint Wald test over pre-treatment event-time coefficients in a two-way fixed-effects event-study regression.Source: §3.2 · p. 3
3.3. Performance Metrics
- Each design cell uses 1,000 Monte Carlo replications, or 500 in the pretrend exercise, and reports power, bias, RMSE, and 95% confidence-interval coverage.The pretrend module also computes the unconditional-versus-conditional rejection gap.Source: §3.3 · p. 3
3.4. Validation
- All five pre-flight unit tests for implementation fidelity pass within their preregistered tolerances.Table 2 summarizes the five implementation-fidelity targets.Source: §3.4 · p. 3, Table 2 · p. 3
4. DiD Results
Across the DiD results, modeling known serial correlation can recover power lost in balanced panels, but attrition, staggered timing, and pretest selection remain outside closed-form power calculations. The simulations also show that inference depends on the attrition mechanism, estimator, and panel setting.
4.1. Sample-Size Scaling Under Serial Correlation and Staggered Adoption
- With 30% attrition, an AR(1)-aware plan still overstates power by roughly 9 pp at N = 200 and 11 pp at N = 500; serial-correlation losses are recoverable in balanced panels.The AR(1)-aware baseline uses the true ρ and DGP; attrition, staggered-timing bias, and pretest selection remain invisible to closed-form formulas.Source: §4.1 · p. 4, Table 3 · p. 4
- With five pre-periods and true ρ = 0.3, naive estimation gives ˆρ ≈ −0.12, while a first-order correction gives ≈0.13 and recovers roughly half the power shortfall.The corrected residual gap is 4–26 pp; bias-aware estimation or longer pre-periods is needed to avoid systematically overstating power.Source: §4.1 · p. 4, Table 4 · p. 4
- Under staggered heterogeneous effects, reference cohort-aware estimators track the cohort-share ATT of 0.23–0.30 SD, while TWFE attenuates it by ≈−0.02 SD.The apparent estimator ordering against base δ is a benchmark mismatch; simplified Callaway–Sant’Anna also understates the ATT by ≈0.03 SD through equal-weight aggregation.Source: §4.1 · pp. 4–5
4.2. Panel Attrition and Effective Sample Size
- At N = 500 and ρ = 0.3, exogenous 30% attrition lowers power from 0.822 to 0.742, while treated-arm outcome-dependent attrition yields ≈−0.05 SD attenuation bias and power of 0.44.Treatment-status-differential but exogenous attrition remains nearly unbiased; outcome-dependent selection requires design or estimation remedies rather than a power correction.Source: §4.2 · p. 5
4.3. Parallel-Trends Pretesting and Conditional Inference
- At N = 1,000, the pretrend test detects violations of 0.05 SD/period in 63% of replications, yet conditioning on passage leaves a 0.249 false-positive rate at a 0.10 SD/period violation.The unconditional rejection rate at that violation is 0.717; selection of replications whose sampling noise masks pre-trends explains the conditional error.Source: §4.3 · p. 5
4.4. Confidence-Interval Coverage
- In staggered homogeneous-effect cells, the simplified Callaway–Sant’Anna analytic interval averages 0.906 coverage and falls below 0.90 in one third of cells.Coverage is flat in N and rises from 0.856 at T = 6 to 0.934 at T = 24; TWFE is nominal at 0.957 and Sun–Abraham is 0.937 in the same cells.Source: §4.4 · pp. 6–7
4.5. Supplementary DiD Robustness Modules
- Under continuous treatment, binary TWFE has bias ≈0.101, RMSE ≈0.143, and power ≈0.491, while absorbing and non-absorbing designs remain near-unbiased with coverage ≈0.95–0.96.Supplementary estimator comparisons were restricted to specifications implemented faithfully; imputation, matrix-completion, and causal-ML comparisons were left for follow-up work.Source: §4.5 · p. 7
4.6. Empirical Anchor: Classic DiD Design
- For the 54-firm, three-year job-training design, i.i.d. power is 0.291, versus mapped achieved power of 0.067 for TWFE, 0.097 for Sun–Abraham, and 0.134 for Callaway–Sant’Anna.The nearest-grid mapping is coarse because the panel’s three periods fall below the simulation grid’s minimum T = 6, so part of the gap cannot be tightly bounded.Source: §4.6 · p. 7
5. IV Results
The IV analysis separates first-stage relevance from exclusion and examines power, bias, and size across instrument strength, sample size, and direct instrument effects. Its results show that small-effect power can remain low despite large samples, while exclusion violations can make larger samples more confidently biased.
5.1. Power as a Function of First-Stage Strength and Sample Size
- IV power never exceeds ≈0.19 at β = 0.1, even with F = 100 and N = 10,000; at F = 10–20, N = 1,000 yields below 0.11.The fixed-F grid isolates instrument strength; with a fixed instrument coefficient, F grows with N and power rises.Source: §5.1 · p. 7, Figure 3 · p. 8
5.2. Two Failure Modes: Weak Instruments versus Exclusion Violations
- At F ≥50, 2SLS bias is in the 10^-3 range, while weak-instrument bias at F = 5 attenuates with N; exclusion bias instead grows with N under fixed F.For γ = 0.05, exclusion bias rises from 0.052 at N = 100 to 0.398 at N = 10,000.Source: §5.2 · pp. 7–8, Figure 3 · p. 8
5.3. Size Properties of Robust 2SLS Under Weak Instruments
- Robust 2SLS size never exceeds 0.069 at F = 5 and is indistinguishable from nominal for F ≥20; weak instruments primarily reduce power and increase first-stage bias.Source: §5.3 · p. 8
5.4. Empirical Anchor: Classic IV Design
- The Card IV design’s i.i.d. power is 0.614 versus achieved power of 0.060; exact-parameter replication bounds the grid-coarseness contribution at about one percentage point.The comparison uses β = 0.119, F = 14.49, and N = 3,010; with a fixed instrument, larger N would raise F and power.Source: §5.4 · p. 8
6. Discussion
The discussion turns the simulation findings into evidence-based design guidance, states boundaries of the simulated setting, and outlines extensions and reproducibility details.
6.1. Implications for Study Design
- Estimate ρ from unit-demeaned residuals with the short-pre-period correction, or report a power curve over ρ ∈[0, 0.7]; model attrition and staggered timing directly.The companion lookup tool’s nine-dimensional cells are preferred to two-dimensional slices for those design frictions.Source: §6.1 · p. 9
- Reference Sun–Abraham and Callaway–Sant’Anna implementations are approximately unbiased for cohort-share ATT, while TWFE attenuates it by ≈ 0.02 SD.Simplified reimplementations can invert estimator rankings; report TWFE alongside a cohort-aware estimator and state the estimand.Source: §6.1 · p. 9
- At F = 10 and N = 500, IV power is < 0.11; exclusion bias is not cured by sample size, and binary pretesting inflates conditional Type I error.The discussion recommends reporting achieved power and an unconditional estimate with sensitivity analysis rather than using a pass/fail pretest gate.Source: §6.1 · p. 9
6.2. Limitations
- The simulated DGP excludes multi-way error dependence, unit-specific trends, and seasonality, making the isolated irreducible gap a conservative lower bound for messier panels.The attrition grid also omits richer dynamic selection, including anticipation and selection on effect size.Source: §6.2 · p. 9
- The study does not evaluate parallel-trends sensitivity analyses, uses 1,000 replications per cell, and treats the AR(1)-aware baseline as an oracle.The DiD empirical anchor lies outside the grid in T, so its nearest-cell mapping has unbounded coarseness.Source: §6.2 · p. 9
6.3. Next Steps: Toward IS Research Methodology for the 2030s
- Next steps include richer failure modes, reference-estimator inference across the grid, uncertainty and sensitivity features in the lookup tool, and a community-contributed living benchmark.The authors expect empirically calibrated simulation evidence to displace closed-form power formulas as the default IS justification standard.Source: §6.3 · p. 9
6.4. Reproducibility and Computational Cost
- The full grids required roughly four days on a 14-core consumer laptop, while a single cell reproduces in minutes and the lookup tool avoids re-simulation within the released grid.The pipeline uses per-cell hash-derived seeds for independent reproducibility.Source: §6.4 · p. 9