Source-linked AI summary
CausalVerify: An Execution-Grounded Benchmark for LLM Causal Inference Workflows
Yonghong Zhang, Ricardo Correia, Isabel M. Parra, Yong Xie
TL;DR
LLM causal benchmarks often assess method descriptions or code execution without verifying recovery of the target causal estimate. CausalVerify separates realistic paper interpretation from executable verification using published papers, fixed-seed synthetic workflows, and calibration reports. Across the evaluated panel, execution-grounded correctness varied widely and was more closely aligned with execution than with text-direction agreement, within a limited standardized scope.
Problem
Existing causal-inference benchmarks usually score textual reasoning or whether code runs, not whether the executed workflow recovers the target causal estimate.
Method
CausalVerify pairs 259 published economics papers for text agreement with 100 fixed-seed synthetic CSV scenarios for executable R workflows, canonical-estimate checks, and retrospective confidence assessment.
Results
Across seven LLMs, L2b+ pass rates range from 10% to 88%, while execution ranking aligns strongly with L2b+ correctness and text-direction ranking does not.
Takeaways & Limitations
Evaluating quantitative causal agents should validate the target quantity because successful execution alone does not establish scientific correctness.
Takeaways & Limitations
The benchmark is confined to standardized single-shot workflows in four design families under the evaluated R backend and does not measure general causal inference.
Abstract
from arXiv · showhide
Existing causal-inference benchmarks for LLMs mostly score method descriptions or whether generated code runs, not whether the executed workflow recovers the target causal estimate. CausalVerify studies this verification problem for structured econometric causal-estimation workflows by separating realistic interpretation from verifiable computation. It pairs 259 published economics papers (reconstructed research question, data description, institutional context) with 100 fixed-seed synthetic scenarios that realise CSV datasets for difference-in-differences, event study, instrumental variables, and regression discontinuity designs. Experiment A (real-paper text agreement) scores method-family and direction agreement against four-LLM consensus labels. Experiment B (synthetic execution) runs model-written R code and checks whether the extracted treatment-effect estimate matches a canonical estimator on the same realised dataset; this execution-grounded correctness layer is L2b+, distinct from L2b, which records only whether the code executes. A calibration arm asks whether self-reported confidence separates correct from incorrect workflows. On Experiment B, seven LLMs reach L2b+ pass rates of 10% to 88% at the default 50% tolerance, and 66 of the 426 workflows that execute (15.5%) return a wrong estimate. Execution ranking (L2b) agrees with L2b+ far better than text-direction scoring (L4): Kendall $τ=0.81$ and Spearman $ρ=0.93$, versus Kendall $τ$ between $-0.20$ and $0.10$ for L4. Llama-3.3-70B-Instruct shows the same qualitative gap, and reported confidence does not reliably separate correct from incorrect workflows. The claims are confined to standardized single-shot workflows in these four design families under the evaluated R backend and model panel; the benchmark does not measure general causal-inference ability. Code, data, cached outputs, and a datasheet are released.
1 Introduction
CAUSALVERIFY addresses a gap in LLM causal-inference evaluation by checking whether executed workflows recover target estimates, not merely whether methods, signs, or code appear correct. It separates realistic paper interpretation from executable verification across complementary benchmark components.
- Valid empirical causal claims require alignment among the estimand, identification argument, estimator, executable code, and numerical result.
- A model can name the right design and produce runnable code while still estimating the wrong coefficient.
- CAUSALVERIFY uses published papers for text-level method-family and direction agreement, not verified causal correctness.
- Synthetic fixed-seed datasets enable executable checks of model-written R workflows against canonical treatment-effect estimates.
- The benchmark separates textual interpretation, code executability, numerical coefficient recovery, and self-assessed confidence.
2 Related Work and Positioning
CAUSALVERIFY complements causal-reasoning, code-generation, and broader science-agent benchmarks by externally verifying the executed causal estimate. Its positioning emphasizes layered verification rather than a single causal-ability score.
- Existing causal benchmarks commonly evaluate textual direction, counterfactuals, graphs, interventions, or identification rather than the executed causal estimate.
- Code benchmarks test execution or functional tests, whereas CAUSALVERIFY checks whether executed analysis code recovers a benchmark-defined target quantity.
- CAUSALVERIFY complements CauSciBench by separating real-paper text agreement, controlled-DGP execution verification, and retrospective calibration.
- Its unit of verification is generated and executed R code on controlled DGP data, with the reported coefficient compared against a realised-data canonical estimate.
- Table 1 compares evaluation affordances, not overall benchmark quality; check marks indicate explicit mechanisms rather than superiority.
3 Benchmark Construction and Controls
The benchmark combines realistic paper contexts, controlled synthetic execution tasks, and retrospective confidence assessment because no single setting provides both ecological realism and executable reference estimates. Its controls fix key data, estimator, and evaluation components while limiting claims to four workflow families.
- Benchmark Construction and Controls: Real papers test context parsing through reconstructed research questions, data descriptions, and institutional contexts, while synthetic DGPs provide executable coefficient targets.
- Scope: The benchmark evaluates downstream design recognition through estimation and reporting, not problem formulation, data acquisition, or iterative robustness analysis.
- Exp A: real-paper context: Exp A includes 259 published economics papers and 1,813 model outputs for text-level method-family and direction agreement.
- Benchmark Construction and Controls: Exp B contains 100 fixed-seed scenarios: 30 DID, 24 Event Study, 24 IV, and 22 RDD tasks.
- Calibration: self-assessment: Calibration uses retrospective verbalized confidence on frozen Exp B outputs and compares those reports with L2b+ correctness.
- Controls and claim boundaries: The synthetic scenarios fix templates, treatment-effect signs, sample sizes, seeds, realised CSV files, and canonical estimators to reduce evaluation circularity.
4 Evaluation Layers
CAUSALVERIFY distinguishes workflow existence, code parsing, execution, numerical agreement, and text-agreement diagnostics. L2b+ compares extracted estimates with realised-data canonical estimates using an intentionally broad, audited tolerance.
- Evaluation layers: L1 checks output existence, L2a checks for parseable R code, L2b executes the code, and L2b+ checks numerical agreement with the canonical estimate.
- L2b+ correctness: L2b+ passes when the extracted treatment-effect estimate matches the realised-data canonical estimate within relative error τ = 0.50.
- L2b+ correctness: Near-zero canonical estimates with |β̂c| < 10^-9 are marked unscoreable rather than used as divisors.
- L2b+ correctness: Event Study scoring freezes accepted post-event windows and scale conversions before model comparison.
- Supporting metrics: L3 and L4 measure method-family and direction agreement in Exp A, serving as diagnostics rather than verified causal-correctness metrics.
- Scorer validation: A separate extraction judge identifies the reported treatment-effect estimate from executed stdout before deterministic numerical comparison.
- Threshold choice: The 50% threshold allows finite-sample and implementation variation while rejecting estimates sufficiently distant to change practical interpretation.
5 Results
Results distinguish textual interpretation, executable code, coefficient correctness, and self-assessed confidence. Execution-grounded correctness separates runnable but wrong workflows, aligns more closely with execution ranking than text-direction agreement, and remains weakly reflected in retrospective confidence.
- Text agreement: L3 and L4 real-paper agreement are text diagnostics rather than executable correctness endpoints, with L4 affected by reference-label noise.L3 is scoreable for 187/259 papers and L4 for 92/259 papers; human agreement with consensus direction labels is only fair, with Cohen’s κ = 0.294.
- Execution-grounded correctness: 10–88%: final L2b+ pass rates span the seven primary commercial models, while 66 of 426 executed workflows return wrong estimates.The wrong-estimate share remains 14.1%–27.2% across the 10%–100% tolerance sweep.
- Execution-grounded correctness: Kendall τ = +0.81 and Spearman ρ = +0.93: L2b execution ranking strongly matches L2b+ correctness ranking across seven models.By contrast, L4 direction agreement has Kendall τ ∈[−0.20, +0.10] with Exp B L2b+ ranking.
- Robustness: L2b+ rankings persist across tolerance sweeps because stronger models concentrate executed estimates near zero while weaker models show larger-error tails.Stricter cutoffs lower absolute pass rates but preserve the separation between text-level and execution-grounded evaluation.
- Calibration: Retrospective confidence does not reliably distinguish correct from incorrect workflows, even when L2b+ accuracy is high.The confidence gap is the mean numerical confidence on correct outputs minus confidence on incorrect outputs; small gaps indicate weak triage value.
- Robustness: Llama-3.3-70B-Instruct reproduces the execution–correctness gap, reaching L2b = 41% and final ES-aware L2b+ = 20%.Adding Llama as an eighth model lowers Kendall τ to 0.714 while preserving the qualitative gap.
6 Reproducibility and Human Audits
The release preserves a frozen, reproducible audit state, while blinded human audits assess both reference-label ambiguity and coefficient-extraction reliability.
- The archived build includes reconstructed paper fields, consensus labels, model outputs, synthetic datasets, canonical estimates, calibration records, hashes, and cached responses.Deterministic scoring can be rerun without new LLM calls, although commercial APIs are not bit-reproducible.
- Human annotation was conducted blindly on 30 papers and did not alter frozen labels, prompts, outputs, or scoring thresholds.The audit was designed as an ambiguity check rather than as training or tuning input.
- 90.9% numeric agreement and 88.6% L2b+ pass/fail agreement were found in the 50-cell coefficient-extraction audit.The audit validates the measurement instrument but does not change headline L2b+ rates.
7 Limitations
The benchmark’s conclusions are bounded by ambiguous text references, synthetic structured tasks, a fixed R backend, retrospective calibration, a limited model panel, audited measurement choices, and single-shot evaluation.
- Reference labels: Exp A consensus labels are imperfect reference labels, with 47.6% human direction agreement and structural overlap between the consensus pool and evaluated panel.Published-paper familiarity from pretraining or public metadata also cannot be fully ruled out.
- Synthetic scope: Exp B measures textbook workflows on structured synthetic data rather than full empirical research, excluding literature review, data cleaning, ambiguous design selection, and iterative robustness analysis.Matching, propensity-score, doubly-robust, mediation, and heterogeneous-effect workflows are outside the benchmark.
- Backend: Fixing R as the execution backend makes L2b+ backend-specific rather than language-invariant causal ability.A Python pilot reproduced the failure mode but cannot establish its magnitude across other environments.
- Calibration: Retrospective calibration tests one confidence-elicitation prompt and does not establish that all calibrated interfaces fail.Coverage is incomplete for some models.
- Model panel: The model panel does not characterize the full open-source ecosystem, and rank correlations are descriptive for the evaluated systems.Only one open-weights model is included as a cross-vendor robustness check.
- Measurement: L2b+ depends on canonical specifications, tolerance choices, scale conversions, and coefficient extraction, so individual-cell mistakes remain possible.Frozen scorer rules, decision records, and a human audit mitigate but do not eliminate measurement uncertainty.
- Interaction: Single-shot, non-interactive scoring measures standardized workflow reliability rather than performance improved through human-in-the-loop debugging or iterative robustness analysis.The pass/fail oracle is unavailable in ordinary real-world use.
8 Conclusion
CAUSALVERIFY argues that executed workflows, not plausible text or merely runnable code, should be evaluated against a specified causal target. Within its defined scope, execution-grounded correctness is more informative than text-direction agreement, while confidence is not reliably discriminative.
- Across seven LLMs, L2b+ pass rates range from 10% to 88%, while execution ranking tracks reference-agreement better than text-direction agreement.The benchmark therefore treats executed workflow verification as the central unit for structured causal-estimation evaluation.
- Evaluation should validate the target quantity rather than use successful execution or plausible text as a correctness proxy.The conclusion is limited to the four design families, R backend, and single-shot protocol evaluated here.
A Reproducibility Details
The reproducibility materials document the archived build, audit provenance, frozen artifacts, and supporting annotation procedures.
- Appendices document frozen artifacts, ambiguity auditing, execution robustness diagnostics, coefficient-extraction auditing, and supplementary pilots.These materials provide the supporting checks behind the benchmark’s reported claims.
- The audit tag v11-freeze-2026-04-29 preserves the provenance trail for the archived build.The PDF checksum is documented in the submission-build summary.
- The archived build is presented as a paper-length rendering of the same frozen claims.The full audit state remains available for inspection.
- The labeling protocol includes a blinded 30-paper ambiguity audit with a documented replacement for an excluded appendix-only PDF.The replacement was drawn from an unsealed sample manifest containing paper ID, difficulty tier, and PDF path.
C Exp B Robustness Details
Exp B robustness checks show that execution-grounded rankings and the executed-but-wrong finding persist across model panels, design families, and tolerance choices, while coefficient extraction remains an audited measurement step.
- Scoring and measurement: L2b+ compares each model-reported estimate with a fixed realised-data canonical estimate, distinct from the structural DGP parameter.The canonical estimator is a benchmark-defined reference path, not the only scientifically defensible analysis.
- Scoring and measurement: Conditional P(L2b+ | L2b) measures whether executed code computes the right coefficient, separating correctness from mere execution.This conditional metric addresses the structural association between L2b and L2b+.
- Failure diagnostics: 340 primary-model cells were non-L2b+ failures, including 230 execution failures and 44 no-code cases; executed-but-wrong behavior varied substantially by model.Conditional pass rates were at least 93% for Opus, Sonnet, and Gemini, versus 22.2% for Kimi and 79.5% for GPT-4o.
- Ranking robustness: Leave-one-model-out correlations remain Kendall 0.733–0.867 and Spearman 0.886–0.943; adding Llama-3.3-70B-Instruct yields Kendall τ = 0.714 and Spearman ρ = 0.881.The primary leaderboard remains the seven-model panel.
- Ranking robustness: Removing any design family leaves the L2b+ ranking within Kendall τ ∈[0.81, 1.00] of the full panel and preserves the reported top-two and bottom-two sets.The comparison uses the full-panel ranking against each reduced-panel ranking.
- Tolerance robustness: The model ranking is identical at 10%, 25%, 50%, 75%, and 100% relative-error tolerances.At the default 50% tolerance, 66 of 426 executing workflows are outside the band; across the sweep, the counts are 116, 79, 66, 61, and 60.
D RID Pilot Ablation
The RID pre-commitment experiment is a small descriptive audit rather than headline evidence: it reruns ten papers per model and shows that text-level outcomes can change in either direction without statistically significant individual comparisons.
- Pilot scope: Experiment C reruns a 10-paper subset per model under RID pre-commitment and is excluded from headline claims because the sample is too small for model-level inference.It is retained as an audit artifact.
- Interpretation: The pilot illustrates that pre-commitment can change text-level L3/L4 outcomes in either direction, without establishing a main benchmark effect.The pilot is not used for the headline benchmark claims.
- Pilot scope: The RID pilot compares RID OFF and RID ON pass rates for L3 strategy and L4 direction, with no individually significant comparison.The figure is descriptive only.
- Supporting diagnostics: Additional diagnostics support the main Exp B conclusions through relative-error profiles, calibration reliability diagrams, and alternative L4 scorer analyses.These appendix diagnostics support, but do not replace, the central correctness argument.
- Supporting diagnostics: Alternative frozen L4 scorers produce unstable pass rates across models, while retrospective confidence does not reliably separate correct from incorrect workflows.The calibration diagnostic reports 95% bootstrap confidence intervals on ECE from 1,000 within-model resamples.