Source-linked AI summary
CausalArena: Benchmarking Causal Discovery in the Foundation Model Era
Zi-Rong Li, Si-Yang Liu, Tian-Zuo Wang, Han-Jia Ye
TL;DR
Causal discovery evaluation is difficult to compare because SCMs, protocols, and generators vary, while pretrained models may overlap with test environments. CausalArena unifies synthetic, semantic operational, and formula-grounded SCMs under an evolvable common protocol with real-world checks. Experiments show substantial ranking shifts across families and settings, so strong performance in one regime does not reliably transfer to others.
Problem
Causal discovery benchmarks vary substantially, and pretrained-model scores may reflect overlap between pretraining environments and evaluation SCMs.
Method
CausalArena combines synthetic, semantic operational, and formula-grounded scientific SCMs under a shared observational and interventional protocol, supplemented by public real-world tables.
Results
Rankings shift substantially across SCM families and protocols, with no method dominating every slice and strong synthetic performance not reliably transferring to semantic, scientific, or real settings.
Takeaways & Limitations
Benchmark diversity, slice-level diagnosis, and attention to pretraining–evaluation overlap are central to interpreting causal-discovery foundation-model results.
Takeaways & Limitations
Released SCMs may eventually enter future training corpora, so overlap cannot be eliminated permanently through public evaluation alone.
Abstract
from arXiv · showhide
Causal discovery aims to uncover causal structures from data and is fundamental to scientific reasoning and intervention-based decision making. Its evaluation relies heavily on structural causal models (SCMs), which specify a causal graph together with the mechanisms that generate data, yet existing studies differ substantially in graph families, mechanisms, and evaluation protocols. The emergence of causal discovery foundation models (CDFMs) further complicates evaluation: performance may reflect not only causal discovery ability, but also overlap between pretraining environments and test SCMs, making results on fixed synthetic benchmarks difficult to interpret. We introduce CausalArena, a unified and evolvable benchmark for causal discovery under a common protocol. Synthetic SCMs supply controlled breadth over structures and mechanisms; semantic operational SCMs provide human-auditable, semantically grounded environments beyond standard synthetic generators; and formula-grounded SCMs test discovery under explicit scientific mechanisms. Public real-world datasets provide an additional external-validity check. Experiments across classical, neural, and pretrained methods reveal substantial ranking shifts across SCM families and protocols, showing that strong performance in one benchmark regime does not reliably transfer to others. These results highlight benchmark diversity and pretraining--evaluation overlap as central challenges for evaluating causal discovery in the foundation model era.
1 Introduction
Causal discovery evaluation varies across SCMs, protocols, and generators, while foundation-model scores may also reflect pretraining–evaluation overlap. CausalArena addresses these issues with diverse, grounded SCM families under a common, evolvable protocol, revealing substantial cross-family ranking shifts.
- Existing evaluations differ in graph families, mechanisms, noise, dimensions, sample sizes, interventions, thresholds, and aggregation, limiting direct comparison across studies.
- Pretrained causal discovery performance can reflect overlap between pretraining environments and evaluation SCMs, not only structure-learning ability.
- CausalArena emphasizes breadth, freshness, grounding, and diagnosability, including evolving evaluation pools for future foundation-model assessment.
- CausalArena combines synthetic, semantic operational, and formula-grounded scientific SCMs with a shared observational and interventional protocol and public real-world checks.
- The benchmark contains 1,200 executable SCM specifications spanning 1,000 synthetic, 100 semantic operational, and 100 formula-grounded scientific environments.
- Rankings shift substantially across Synthetic, Semantic, Formula, and real-data settings, with no method dominating every slice and synthetic strength not reliably transferring elsewhere.
2 Related Work
Prior causal-discovery resources provide either scalable synthetic evaluation or semantic, scientific, and real-world grounding, but usually not both under a shared protocol. CausalArena is positioned to unify these complementary strengths for cross-method comparison.
- Classical, amortized, and pretrained causal-discovery methods use differing assumptions and learning principles, motivating broad evaluation across method families.
- Synthetic SCM frameworks enable scalable evaluation with known graphs, mechanisms, noise distributions, and observational or interventional samples.
- Grounded resources add semantic, scientific, physical, or real-data context, but often target narrower tasks or domains than full multivariate DAG recovery.
- Existing resources distribute synthetic breadth, grounding, shared protocols, and joint observational–interventional evaluation across separate benchmarks.
3 Benchmark Formulation and Design
CausalArena defines causal discovery as recovering a ground-truth adjacency matrix from observational or interventional SCM data and evaluates methods through common interfaces and factorized scores. Its complementary SCM families jointly provide breadth, grounding, freshness, and diagnosability while supporting evolving evaluation for foundation models.
- 3.1 Task and Evaluation Formulation: The task maps observational or interventional data from an SCM to an estimated directed adjacency matrix, scored against ground truth with F1 and SHD.
- 3.1 Task and Evaluation Formulation: Classical and per-dataset neural methods operate on each dataset, whereas CDFMs pretrain reusable parameters on collections of environments before inference.
- 3.2 Design Requirements: Scores are defined relative to benchmark slices and retain construction factors such as graph family, mechanisms, noise, dimensions, interventions, and domains for diagnosis.
- 3.2 Design Requirements: The benchmark requires breadth, freshness, grounding, and diagnosability because fixed benchmark scores may not transfer across environments or remain outside future pretraining corpora.
- 3.2 Design Requirements: All families compile into a shared executable interface for observational and interventional evaluation, with scores broken down by family, mechanisms, protocols, domains, and real-data checks.
- 3.3 Three Complementary SCM Families: Synthetic SCMs provide regenerable factor grids and diagnosis, while semantic operational SCMs add human-auditable meaning and newly authored environments for freshness.
- 3.3 Three Complementary SCM Families: Formula-grounded scientific SCMs encode named equations and validity ranges, enabling mechanism-level and equation-level diagnosis alongside the other families.
- 3.4 Evolvable Evaluation for Foundation Models: Evolving the SCM pool under fixed interfaces and reporting training disclosure helps interpret overlap, although released SCMs may later enter future training corpora.
4 Benchmark Implementation
CausalArena implements three complementary SCM families under shared evaluation interfaces: synthetic configurations provide controlled breadth, while semantic and formula-grounded scenarios add operational and scientific grounding. The arena combines executable construction, family-specific validation, and extensibility across 1,200 SCM specifications.
- CausalArena contains 1,200 executable SCM specifications spanning 1,000 synthetic, 100 semantic operational, and 100 formula-grounded scientific scenarios.The public package releases half of each family, reserving the remaining audited SCMs for held-out leaderboard evaluation and progressive versioned updates.
- Synthetic SCMs: Synthetic SCMs vary graph topology, dimensions, mechanisms, distributions, dependencies, interventions, and difficulty while retaining exact ground-truth structures.The pool spans dimensions 10, 20, 30, 50, and 100 across multiple graph, mechanism, distribution, dependency, and difficulty families.
- Semantic Operational SCMs: Semantic SCMs encode interpretable operational processes whose labeled variables, justified edges, and executable interventions support grounded and extensible environments.The semantic pool covers 100 scenarios across 10 operational domains, with 17–25 variables and 25–67 directed edges per scenario; new scenarios can be added without changing the evaluation interface.
- Examples: Released examples show semantic and formula SCMs as directed graphs: wildfire smoke has 20 variables and 26 edges, while barometric pressure reduction has 16 variables and 18 edges.The examples connect operational measurements, interventions, calibration steps, and mechanistic computations within executable SCMs.
- Formula-Grounded Scientific SCMs: Formula-grounded SCMs anchor selected mechanisms in explicit scientific equations, compute formula nodes exactly, and validate units, ranges, orientations, and residuals.The family contains 100 scenarios across 10 scientific and engineering domains, with 16–25 variables and 18–50 directed edges per scenario.
- Validation and Release: Semantic and formula-grounded SCMs use staged human and LLM-assisted review, whereas synthetic SCMs rely on deterministic programmatic checks and all families undergo diversity control.Reviews and audits address graph validity, executable mechanisms, measurement realism, intervention meaning, numerical behavior, coverage, redundancy, unsupported edges, and low-information variables.
5 Results and Observations
Under a shared protocol, causal-discovery rankings shift substantially across SCM families, real datasets, sample sizes, intervention designs, and runtime considerations. These results show that pooled or single-regime scores can obscure transfer, protocol sensitivity, and efficiency trade-offs.
- 5.1 Main Results: FoundCause leads supported observation-only slices, GIES is competitive with interventions, and TabCausal is strong with interventional evidence, but no method wins every slice or metric.FoundCause reaches F1 0.62/0.63/0.52 on Synthetic/Semantic/Formula; GIES reaches 0.57 on Semantic obs+int; TabCausal reaches 0.53 on Formula obs+int.
- 5.2 Real-data Check: Real-data rankings reverse the synthetic pattern: CDFM leads observation-only F1 at 0.39, TabCausal leads observation-plus-intervention at 0.46, and FoundCause falls to 0.21.Per-dataset scores vary widely, so aggregate means and standard deviations can be dominated by a few tables.
- 5.3 Sample Size and Intervention Protocol: Most sample-size gains occur from n = 100 to n = 1,000, after which curves flatten, while Arrow and CauScale decline at n = 10,000.FoundCause remains highest from 0.43 at n = 100 to 0.66 at n = 10,000, but the suite’s fixed d = 30 SCM composition constrains interpretation.
- 5.3 Sample Size and Intervention Protocol: Intervention protocols affect methods unevenly: TabCausal and SDCD are stable, whereas IGSP drops under multi-target, dense mixed, and high-intervention-fraction settings.IGSP’s F1 changes are −0.27, −0.15, and −0.09 respectively, with F1 standard deviation 0.100 versus 0.009 for TabCausal and 0.010 for SDCD.
- 5.5 Runtime and Resource: The accuracy–cost frontier favors different trade-offs: Arrow and AVICI are cheap, CDFM and TabCausal raise F1 at similar cost, and FoundCause attains the highest supported observation-only F1 at higher Tgraph.The comparison omits pretrained-model load time because large runs can amortize it; documented pretraining coverage is uneven across the benchmark.
6 Analysis
Causal discovery methods respond differently to graph structure, mechanisms, noise, scale, and pretraining exposure. These sensitivities produce substantial out-of-distribution variation and domain-specific strengths, limiting the interpretability of pooled rankings.
- Graph structure: Bipartite-layered graphs favor CDIS, GIES, and FoundCause, whereas continuous-optimization methods and AVICI are relatively stronger on sparse branching families.The bipartite-layered interactions are +0.192, +0.120, and +0.098 for CDIS, GIES, and FoundCause, versus −0.108, −0.089, and −0.089 for DAGMA, NOTEARS, and AVICI.
- Scale effects: FoundCause remains strongest under increasing dimension, while CauScale improves and several pretrained methods decline sharply by d = 100.FoundCause falls from 0.67 at d = 10 to 0.56 at d = 100; CauScale rises from 0.35 to 0.48, while Arrow, CDFM, and AVICI reach 0.21, 0.27, and 0.16.
- Mechanism and noise effects: Mechanism and noise families separate methods unevenly: nonlinear tree-based mechanisms favor AVICI, CDFM, and NOTEARS-MLP, while mixture-outlier noise favors LiNGAM.The tree-based row gives +0.100, +0.086, and +0.082 to AVICI, CDFM, and NOTEARS-MLP; mixture-outlier noise gives LiNGAM +0.126 and SDCD −0.097.
- Pretraining-exposure OOD analysis: Holding out undocumented pretraining categories preserves FoundCause and TabCausal as the top two but reveals markedly different out-of-distribution gains.FoundCause has about +60% overall OOD gain, followed by TabCausal at +22% and CauScale at +10%, while Arrow and AVICI are negative at −17% and −25%.
- Semantic and formula behavior: Semantic and formula performance depends on causal-story structure: local screening and inspection cascades suit conditional-independence search, while shared parameters, multiple outputs, and derived readouts remain difficult.PC recovers local screening and inspection cascades better than adaptive or behavioral processes, and formula methods handle low-coupling systems better than equations with shared parameters or multiple outputs.
7 Discussion
CausalArena is designed for benchmark development when closed-loop validation is unavailable, combining executable SCMs, versioned releases, and training-context disclosure. Its current scope is tabular structure recovery with known directed ground truth, while public real-world graphs provide only reference structures.
- Executable SCMs expose graph structure, mechanisms, interventions, and sampling procedures for scalable causal-discovery development when real-system feedback is unavailable.
- CausalArena distinguishes public and reserved SCM subsets and records benchmark versions alongside disclosures of related pretraining or development data.
- Pretraining–evaluation overlap cannot be eliminated permanently, so future evaluations should extend the SCM pool and report benchmark versions with training disclosures.
- The current release targets multivariate tabular causal structure recovery with known directed graph ground truth, while its interface supports future extensions such as latent confounding and neighboring causal tasks.
- Public real-world datasets offer an external-validity check, but their published graphs are reference structures rather than uniquely verified causal ground truth.
8 Conclusion
CausalArena unifies synthetic, semantic operational, and formula-grounded scientific SCMs under shared observational and interventional protocols. Experiments show that method rankings vary across evaluation regimes, motivating versioned and extensible evaluation that accounts for possible pretraining–evaluation overlap.
- CausalArena combines three SCM families under shared observational and interventional evaluation, with public reference suites and an extensible executable interface.
- Rankings vary substantially across SCM families, mechanisms, sample sizes, and intervention settings, so strong performance in one regime does not necessarily transfer to others.
- For pretrained models, benchmark scores should be interpreted in light of possible pretraining–evaluation overlap as methods and foundation models evolve.
A Benchmark Data Details
The benchmark data are organized as audited synthetic, semantic, and formula-grounded SCMs with controlled structural and mechanism diversity. A shared executable workflow and intervention protocol support reproducible evaluation across dimensions, graph families, domains, and sample sizes.
- The synthetic benchmark contains 1,000 SCM configurations crossing 10 graph families with five dimensionalities from d = 10 to d = 100.
- Synthetic graph families include Erdos–Renyi, scale-free, chain, tree, layered, bipartite layered, and small-world structures with controlled density ranges.
- Mechanism and noise-family diversity increases with dimension, while local assignments vary mechanisms, root families, and noise choices across nodes.
- The main intervention split uses 800 observational and 200 interventional rows, with balanced hard do-resampling across dimension-dependent target pools.
- Synthetic SCMs undergo graph- and dataset-level audits checking acyclicity, connectivity-related structure, depth, edge counts, finite values, variable variation, category balance, and boundary mass.
- Sample-size and intervention-protocol suites reuse representative synthetic subsets while varying observational counts or intervention-generation rules.
- Semantic and formula-grounded SCMs are built through staged, auditable workflows in which failed checks return to graph design or executable specification before release.
B.4 Method Settings
Methods use fixed wrapper defaults and, for pretrained or amortized methods, fixed checkpoints across generated-SCM and real-data runs. Results report multiple graph-recovery metrics with deterministic handling of native equivalence-class outputs.
- Methods are generally run once with wrapper defaults and no hyperparameter search on benchmark labels.
- Pretrained and amortized methods use the same named checkpoint or source across generated-SCM and real-data evaluations.
- Directed-edge F1 and SHD are computed after deterministic conversion rules for methods that output equivalence-class objects.
- Reported metrics include F1, precision, recall, SHD, SID, nSHD, AUROC, and AP when supported by the method output.nSHD is computed as SHD/(d(d −1)); hard-graph methods lack AUROC/AP entries.
- Values are reported as mean±standard deviation across heterogeneous graph instances or datasets, with best and second-best values marked within comparable blocks.
C.1 Synthetic Benchmark
The synthetic benchmark reports overall and factor-level results across graph families, dimensions, dependency modes, difficulty components, mechanisms, and noise labels. It also compares methods through a complete observation-only pairwise win-rate matrix.
- The synthetic benchmark includes overall metrics plus decompositions by graph family, dimension, root-dependency mode, difficulty component, mechanism, and noise.
- The pairwise win-rate matrix pools F1 and SHD wins across per-SCM or per-dataset comparisons after averaging repeated runs within each unit.
- Observation-only rows report distinct performance profiles across CDFM, FoundCause, SEA, and TabCausal on the 5k synthetic setting.
- Observation-plus-intervention rows compare classical, neural, and pretrained methods using F1, precision, recall, SHD, nSHD, SID, AUROC, and AP where available.
- Mechanism and noise analyses use compact edge-level F1 views computed from aggregated TP/FP/FN counts.
C.2 Semantic Benchmark
The Semantic benchmark reports complete metrics for semantically grounded environments and analyzes domain-level behavior separately. Its results are summarized over replicate-level rows.
- The Semantic benchmark’s complete metrics are reported in Table 14 over replicate-level rows.
- Domain-level Semantic behavior is analyzed separately in Section 6.3.
- The reported Semantic rows compare observation-plus-intervention methods across F1, precision, recall, SHD, nSHD, SID, AUROC, and AP when available.
C.4 Real-data Benchmark
The real-data benchmark summarizes method-level and dataset-level performance across observation-only and interventional sources. It includes six observation-only datasets and seven interventional sources.
- The real-data evaluation includes six observation-only datasets and seven interventional sources spanning Sachs, PetShop, and Causal Chambers.
- Table 16 provides complete real-data method metrics, while Table 17 lists every method–dataset pair.
- The observation-only table combines native observation-only datasets with observation-only conversions of the seven interventional datasets.
- Observation-only results show varied profiles across CauScale, CDFM, FoundCause, SEA, and TabCausal on 13 datasets.
- Observation-plus-intervention results compare seven-source performance for classical, neural, and pretrained methods, with score availability varying by output type.
C.5 Sample-size Scaling
The sample-size scaling suite evaluates observation-only causal discovery with d = 30 while varying the number of samples, reporting F1 and SHD.
- The suite fixes d = 30 and varies the number of samples in an observation-only synthetic setting.
- The reported metrics are F1 and SHD for each sample-size condition.
- Table 18 reports the complete sensitivity results for this suite.
C.6 Intervention-protocol Sensitivity
The intervention-protocol sensitivity suite compares causal discovery methods across parameter-shift, multi-target, dense-mixed, and high-fraction protocols using multiple structural and predictive metrics.
- The suite evaluates mixed observational/interventional data across parameter-shift, multi-target, dense-mixed, and high-fraction protocols.
- The intervention-protocol results are presented as complete numerical sensitivity metrics, while construction-quality scores are reported separately for the same sampled scenarios.
- The tables report F1, precision, recall, SHD, nSHD, SID, AUROC, and AP when available for each protocol and method.
- Under parameter shift, TabCausal reports the highest listed F1 at 0.52±0.12, while AVICI reports the highest listed AUROC at 0.85±0.08.