Source-linked AI summary
CausalDS: Benchmarking Causal Reasoning in Data-Science Agents
Andrej Leban, Yuekai Sun
TL;DR
Existing benchmarks often separate symbolic causal reasoning from realistic data-science analysis, despite causal workflows requiring structure interpretation, data inspection, estimation, and warranted abstention. CausalDS evaluates these abilities jointly in synthetic SCM-grounded scenes, finding that models largely master structure and estimates but diverge on uncertainty, abstention, and tool use.
Problem
Existing benchmarks often separate symbolic causal reasoning from realistic data-science analysis, while causal workflows require structure interpretation, data inspection, identifiability judgments, estimation, or abstention.
Method
CausalDS generates SCM-grounded synthetic scenes with stories, tabular data, hidden ground truth, observation models, and tasks scored across five causal data-science axes.
Results
Models largely master structure and estimates but diverge on uncertainty, abstention, and tool-use efficiency, with Claude Opus 4.8 closest to a well-rounded causal data-science agent.
Takeaways & Limitations
Causal data-science competence decomposes across capabilities, so evaluation must assess symbolic reasoning, estimation, uncertainty, abstention, and tool use together.
Takeaways & Limitations
The authors identify future needs for larger, more varied exams, deeper failure taxonomies, and targeted stress tests for abstention, quantitative skills, and counterfactual reasoning.
Abstract
from arXiv · showhide
Large language models (LLMs) increasingly act as integrated data-science agents, combining abstract reasoning with advanced tool use. Yet the relevant benchmark landscape largely divides into symbolic causal reasoning benchmarks without realistic data analysis or data analysis benchmarks without a principled causal data-generating structure. Furthermore, existing causal evaluation datasets are often restricted to curated examples from existing sources, with diversity coming from limited templatized variations rather than from systematic generation of novel synthetic causal structures. We introduce CausalDS, a benchmark for evaluating causal reasoning in agentic data-science workflows. Each benchmark instance is a scene consisting of a sampled structural causal model (SCM) with generated observational data and an accompanying synthetic natural-language story grounded in a realistic domain. We optionally ground the composition of the benchmark components in empirical distributions obtained from real-world datasets, thus retaining empirical structure while reducing the "causal parrot" risk through completely synthetic generation. From each scene, we then derive tasks spanning all three of Pearl's rungs, with typical data-science prediction tasks appearing as Rung 1. Most tasks include a data science coding component, where the model typically needs to use several tools to arrive at the final answer due to the frequent presence of imperfect observations, which are generated by an observation model. Additionally, recognizing when a question admits no warranted answer and abstaining is treated as a first-class scored outcome. The benchmark thus jointly evaluates symbolic causal reasoning, data science, uncertainty quantification, abstention, and tool use/coding.
1 Introduction
CausalDS benchmarks causal reasoning in realistic, agentic data-science workflows by combining symbolic reasoning, data analysis, uncertainty quantification, abstention, and tool use. It addresses gaps in isolated benchmarks through synthetic SCM-grounded scenes with private ground truth, natural-language descriptions, tabular data, and deterministic scoring.
- Benchmark scope: CausalDS evaluates symbolic causal reasoning, data-science execution, uncertainty quantification, abstention recognition, and tool-use efficiency together.These dimensions are usually tested separately, but realistic causal analysis requires their interaction.
- Benchmark scope: The benchmark requires agents to interpret causal scenarios, reason over graph structure, distinguish associational, interventional, and counterfactual targets, and analyze tabular data.Data-science execution involves using standard analysis tools to produce estimates and predictions.
- Motivation: Unlike data-science benchmarks lacking hidden causal structure and causal benchmarks operating purely symbolically, CausalDS integrates both settings in an agentic, file-backed environment.The benchmark uses deterministic scoring and evaluates whether capabilities remain distinct across agents.
- Scene generation: CausalDS generates synthetic causal data-science scenes containing hidden causal graphs, instantiated SCMs, observational data, private ground truth, natural-language descriptions, and tabular data.Benchmark composition can be anchored to empirical axes including variable type, graph structure, identifiability, mechanism profile, and observation complexity.
2 Background
CausalDS is grounded in structural causal models and Pearl’s three-rung hierarchy, spanning associational, interventional, and counterfactual reasoning. Its tasks also test identifiability, principled abstention, and code-driven interaction with a sandboxed data-science environment.
- Causal graphs, SCMs, and Pearl’s hierarchy: An SCM assigns each variable a structural equation from its graph parents and an exogenous disturbance, inducing the observational distribution P(V).The same SCM also defines interventional distributions by replacing a structural mechanism and counterfactual quantities across parallel worlds sharing exogenous draws.
- Causal graphs, SCMs, and Pearl’s hierarchy: Pearl’s hierarchy organizes tasks into Rung 1 associational reasoning, Rung 2 interventions P(Y | do(X)), and Rung 3 counterfactual reasoning.Rung 1 includes standard data-science tasks such as prediction.
- Effects and Identifiability: A causal effect is identifiable when its interventional functional can be expressed using observational terms alone, with do-calculus and ID providing a complete decision procedure.The back-door and front-door criteria are classical sufficient conditions for identifiability.
- Effects and Identifiability: Abstaining on non-identifiable estimands is a first-class scored outcome, while counterfactual reasoning captures quantities such as the effect of treatment on the treated that Rung 2 cannot express.Non-identifiable problems may occur wherever non-identifiability is possible.
- Data-science agents: A data-science agent is an LLM that reads benchmark files, executes code, inspects intermediate outputs, and writes answer files in a sandboxed environment.Accordingly, each task combines language understanding, code-driven estimation, and basic tool use.
3 The CausalDS Benchmark
CausalDS constructs benchmark scenes by combining synthetically generated causal structures, structural causal models, imperfect observations, domain-grounded narratives, and tasks spanning Pearl’s three rungs. Its design separates conceptual causal identifiability from measurement difficulty while supporting increasingly complex graphs and realistic verbalization.
- Scene construction: Each scene contains a narrative story, tabular dataset, lightweight schema, task list, and separately stored ground truth with a hidden test split.Scene generation proceeds from causal-graph sampling through SCM instantiation and subsequent benchmark execution.
- Causal structure: Causal graphs begin with canonical motifs and can expand through anchor-based grafting of auxiliary motifs joined by exactly one shared node.Grafting preserves narrative coherence, limits mapping to isolated subgraphs, and supports latent-variable motifs for identifiability and adjustment stress tests.
- Structural causal models: SCMs mix continuous and binary variables through independently selected typed mechanism profiles combining empirical mean-function families with synthetic regimes.Additive continuous noise and single-link binary noise preserve suitability for Monte Carlo evaluation of treatment effects, mediation, and counterfactuals.
- Measurement: The observation layer withholds selected conceptual variables and replaces them with noisy measurement bundles, preserving causal estimands while making numerical estimation harder.Identifiability is evaluated on the conceptual graph rather than the measurement layer; hidden confounders remain unobserved everywhere, whereas measured variables remain part of the scene.
- Verbalization and tasks: An LLM maps abstract nodes to domain-relevant names, optionally seeds nodes with CauseNet, audits mappings, and generates and verifies narratives describing causal relations.Tasks are organized as Rung::family::variant across Pearl’s associational, interventional, and counterfactual hierarchy, with compatibility and identifiability gating for some variants.
4 Related work
Prior work benchmarks real-data causal analysis, formal causal reasoning, or agentic data-science workflows, but CausalDS integrates these ingredients into narrated, SCM-generated scenes with tool-backed deterministic evaluation. Its unified design also includes noisy measurements, Pearl-ladder tasks, and scored abstention for non-identifiability.
- Real-study causal analysis benchmarks: Real-data benchmarks evaluate causal analysis on curated scientific or policy datasets, emphasizing method choice, identification, estimation, uncertainty, or intervention-centered study design.CauSciBench and CausalReasoningBenchmark cover end-to-end analysis, whereas InterveneBench focuses on interventions in real social systems without predefined graphs or structural equations.
- Synthetic causal benchmarks and causal reasoning probes: Synthetic causal benchmarks probe formal reasoning through small graph motifs, graph-centric tests, statistical pitfalls, or executable counterfactuals.CLadder spans Pearl’s ladder with oracle-generated answers; other benchmarks target graph understanding, confounding, Simpson’s paradox, selection bias, or code-mediated counterfactual reasoning.
- CausalDS’s integrated design: CausalDS makes each benchmark unit a narrated scene combining a hidden SCM, domain-grounded variables, a graph-audited free-form story, synthetic SCM-driven data files, and deterministic scoring of tool-backed answers.This scene-level construction distinguishes the benchmark from approaches covering only individual ingredients.
- Agentic data-science benchmarks and causal-agent systems: Agentic data-science benchmarks evaluate code generation, machine-learning engineering, and open-ended data-analysis workflows, often through interactive execution frameworks.These benchmarks motivate CausalDS’s file-backed, tool-using workflow design.
- CausalDS’s integrated design: CausalDS uniquely integrates graph-audited free-form scenes, SCM-generated data files, noisy measurements, complete Pearl-ladder estimand tasks, deterministic scoring, and first-class abstention for non-identifiability.The paper characterizes this as a single generator spanning ingredients otherwise distributed across prior and concurrent work.
5 Main Results
On a 100-scene realistic-composition exam, Claude Opus 4.8 leads aggregate performance, while Rung 2, abstention, imperfect observations, and uncertainty quantification differentiate models. Repeated attempts improve best-case binary performance but reveal substantial worst-case gaps without indicating unstable point estimates.
- Headline results: Claude Opus 4.8 leads the realistic-exam leaderboard with CausalDSScore 0.278 and best or tied-best Pass Rate (82.4%), Med. NRel. Err (0.179), and SNR (0.566).Two frontier reasoning models occupy the top two composite spots, with Gemini 3.1 Pro second at 0.370.
- Failure axes: Rung 2 and uncertainty quantification provide the strongest differentiation: R2 Pass Rate spans 28.6%–92.9%, identification Pass Rate spans 28.6%–100.0%, and 95% ATE coverage spans 20.0%–71.4%.The passage characterizes the coverage results as evidence of model overconfidence, with Claude performing best.
- Failure axes: Content correctness ranges from 81.0%–100.0%, whereas abstention ranges from 18.8%–75.0%, with frontier models leading the abstention slice.GPT-5.5 has the highest abstention pass rate at 75.0%, followed by Claude Opus 4.8 at 62.5% and Gemini 3.1 Pro at 56.2%.
- Observation hardness: Harder observation views make produced estimates substantially worse and additionally cause outright failures to answer, with hardest-view mean errors far exceeding medians for most models.Gemma 4 26B reaches mean NRel. Err 4.59 versus median 0.348, while GPT-5.5 reaches 3.10 versus 0.222; Claude remains controlled at 0.557 versus 0.183.
- Repeated attempts: For binary outcomes, pass@3 reaches 76.5%–91.2% while passˆ3 reaches 50.0%–61.8%, producing a 26–29-point best-case versus worst-case gap.Kimi K2.6 reaches 91.2%, matching the best frontier single-shot rate on the same pool.
- Repeated attempts: Repeated-attempt CausalDS score variance is 7%–27% of the mean, while median per-task NRelErr SD is ≤0.002 across tasks answered in all three runs.The passage attributes score variance to relatively few uncertainty-quantification tasks and abstention decisions rather than benchmark instability.
6 Discussion
CausalDS jointly scores symbolic reasoning, estimation, uncertainty quantification, abstention, and tool-use efficiency under hidden SCMs and corrupted data views. On the presented exam, models largely master structural reading and estimation but diverge on judging estimate quality and whether answers are licensed.
- Benchmark scope: CausalDS scores symbolic causal reasoning, quantitative estimation, uncertainty quantification, epistemic abstention, and tool-use efficiency using hidden SCMs, verbalizations, ground truth, and corrupted data views.Each target includes an identifiability status, while an observation model corrupts the released data view.
- Exam findings: All six models recover structure essentially perfectly and make symbolic identification calls mostly without error, while performance dissociates on epistemic axes and tool use.The frontier–open-weight separation appears on uncertainty quantification, abstention, and tool use/efficiency; frontier models lead abstention outright.
- Future work: Future work should expand exam size and variety, develop deeper trajectory-level failure taxonomies, and add targeted stress tests for abstention, quantitative skills, and counterfactual reasoning.These directions extend the benchmark along the axes it already exposes.
- Interpretation: Models largely master reading the structure and producing estimates, but diverge on knowing how good an estimate is and whether any answer is licensed.CausalDS frames this whole-agent judgment as distinct from isolated symbolic or data-science benchmark performance.
A Appendix · A.1 Code and data availability
The CausalDS repository provides the benchmark’s complete implementation and datasets, supporting reproduction of the main exam and ablation studies.
- A.1 Code and data availability: The repository contains the CausalDS scene-generation pipeline, evaluation harness, and grader.These components comprise the benchmark’s source code.
- A.1 Code and data availability: It provides the entire datasets used for the main exam.
- A.1 Code and data availability: The same datasets also cover the ablations presented in the work.
A.2 Scene generation details
Scene generation uses documented continuous and binary SCM profile registries with varied noise families and binary link mechanisms. Continuous variables draw from Gaussian, Laplace, Student-t, or Gaussian-mixture noise, while binary mechanisms use stochastic Bernoulli outputs.
- Noise families and link functions: Continuous SCM noise families include Gaussian, Laplace, Student-t, and Gaussian mixtures.These families are documented in the continuous SCM profile registry.
- Noise families and link functions: Binary mechanisms produce Bernoulli outputs through logistic, threshold, or noisy-gate links.The binary link mechanisms are documented in the binary SCM profile registry.
- Noise families and link functions: Binary mechanisms have no explicit noise term; the Bernoulli draw supplies stochasticity.Stochasticity therefore arises from sampling the Bernoulli output rather than from a separate noise variable.
A.3 Scene synthesis sub-algorithms
Scene synthesis uses three audited subroutines: PREAUDIT checks CauseNet-seeded graph feasibility, MAPSTAGE assigns variable names with repair, and VERBALIZE generates and verifies the natural-language story. Together, these stages enforce graph plausibility, naming constraints, and consistency between the story and DAG.
- Overall control flow: The top-level synthesis flow invokes PREAUDIT, MAPSTAGE, and VERBALIZE for feasibility checking, graph-variable naming with audit/repair, and story generation with DAG verification.Each subroutine has a defined output, including feasibility verdicts, mappings or failure, and verified stories or failure.
- PREAUDIT: PREAUDIT extracts graph edges and non-edges, then rejects CauseNet seeds when fixed-node non-edges imply unavoidable mainstream direct-cause relationships.It returns a feasibility verdict with confidence and a reason after assessing whether placeholders can be named while preserving plausible graph constraints.
- MAPSTAGE: MAPSTAGE proposes a complete, unique variable mapping under fixed-name constraints and iteratively audits edge plausibility, non-edge independence, and type consistency.It returns the mapping when violations are resolved, or fails when a violation cannot be repaired without changing fixed names.
- VERBALIZE: VERBALIZE drafts a domain-grounded story from the mapping and DAG, then verifies missing variables, missing edges, direction contradictions, and other warning conditions.The process builds yellow-flag context from derived-name markers and restrictive qualifiers before retrying story generation within a configured budget.
A.4 Mapping and narration audit details
The mapping and story auditors use a deterministic preprocessor to flag suspicious variable names and restrictive qualifiers, targeting cases where broad latent constructs are narrated as direct causes of administrative artifacts.
- A.4 Mapping and narration audit details: A deterministic preprocessor scans variable meanings for derived-name markers and restrictive qualifiers, then appends a yellow-flag block directing auditors to suspicious nodes.The method targets a recurring failure mode involving broad latent constructs narrated as direct causes of administrative artifacts.
A.5 Motif catalog
The sampler uses base motif templates as pre-augmentation graph skeletons during grafting, with latent variables marked as gray dashed nodes.
- A.5 Motif catalog: Base motif templates serve as the sampler’s pre-augmentation graph skeletons during grafting, and gray dashed nodes denote latent variables.Figure 5 presents these templates and their latent-variable notation.
A.6 Observation-model implementation details
Observation mechanisms reuse the SCM’s mechanism and noise families while incorporating shifted nonlinearities and optional heteroscedastic noise. Candidate bundles undergo Fisher-information admissibility checks, whereas private recoverability diagnostics characterize attainable measurement-based recovery without determining acceptance.
- Mechanism families and noise: Observation mechanisms use handcrafted linear–nonlinear–interaction forms or small spectral-normalized neural networks, with independently sampled shifts moving tanh cancellation points.The shift c is sampled from U(−1.5, 1.5) per measurement to avoid flat regions near the realized treatment-contrast region.
- Mechanism families and noise: Heteroscedastic noise replaces constant σ with a clamped softplus function σ(z), making the Fisher denominator location-dependent.The supplied passage specifies σ(z) = softplus(w⊤σ z + bσ), clamped to [σmin, σmax].
- Admissibility checks on candidate bundles: Candidate measurement bundles are accepted only when Fisher information clears minimum, average, and lower-tail thresholds across the realized latent-support grid.Outcome measurement bundles use stricter thresholds; failed resampling attempts lead to scene rejection before verbalization.
- Recoverability diagnostics in the private bundle: Private recoverability diagnostics report R2 for continuous variables and AUC for binary variables, including an intrinsic upper bound based on jointly using all measurements.A multi-measurement information-gain gap compares this upper bound with the best single-measurement nonlinear loss.
- Recoverability diagnostics in the private bundle: Recoverability scores are diagnostic summaries rather than acceptance criteria; admissibility remains determined by the mechanism checks.The diagnostics are recorded with per-variable gaps, and the upper bound limits even an oracle’s downstream performance.
A.7 Optional empirical grounding of composition axes
CausalDS remains fully synthetic at the scene level while optionally grounding composition axes in several empirical corpora rather than imitating any single source. The framework uses axis-specific anchors, explicit crosswalks, credibility weighting, and design-time adjustments to construct realistic but controllable benchmark mixtures.
- Grounding strategy: Empirical grounding targets individual composition axes from several corpora, with source weights reflecting how directly each corpus exposes the relevant quantity.The benchmark preserves fully synthetic scene generation while using empirical anchors to resemble real causal-analysis workloads.
- Structural composition: Treatment and outcome types are reduced to a continuous-versus-non-continuous split using row-level causal benchmarks and credibility-weighted source mixtures.Sources include EconCausal, CauSciBench, CausalReasoningBenchmark, and CauSciBench’s QR/textbook source.
- Structural composition: 99.6% of pooled empirical benchmark cases are identifiable, so challenge suites inform represented identification obstacles while empirical sources do not determine non-identifiability frequency.The small empirical remainder is unresolved rather than explicitly labeled non-identifiable.
- Structural composition: Mechanism profiles draw counts only from biochemical-equation and Boolean-rule corpora that expose per-equation or per-rule rows, applying partial-export weights when coverage is incomplete.These sources include BioModels, SABIO-RK, BiodivineBooleanModels, and the Kadelka DesignPrinciples corpus.
- Exam composition: Exam composition combines empirical task-family marginals with a CLadder graph-by-query structural lift, while a uniform deployable-family floor prevents rung-1 or rung-3 families from collapsing to zero mass.The resulting question-type prior is P(Q | S), or Exam Axis 1; output and observation variants are separately conditioned on question type.
A.8 Exam-side difficulty knob
The exam-side difficulty knob d ∈ [0, 1] jointly tilts question types, output variants, and released observation variants, with d = 0.5 neutral and endpoints moving from easier, clean R1 settings toward harder, proxy_hard R3 settings.
- Exam-side difficulty knob: Difficulty d ∈ [0, 1] tilts P(Q | S), P(OutputVariant | QuestionType), and P(obs | Q) across question families, output variants, and observation variants.Question families are indexed by Pearl rung, output variants by within-family difficulty rank, and observation variants by o(obs) ∈ {0, 1, 2}.
- Exam-side difficulty knob: d = 0.5 is a no-op, while d = 0 favors R1, easier contracts, and clean observations, and d = 1 favors R3, harder contracts, and proxy_hard observations.The dial is editorial rather than a frequency claim, and Pearl rung is only a coarse
- Exam-side difficulty knob: β, δ, and γ remain separate, allowing rung, output-variant, and observation pressure to be calibrated independently, with all three defaulting to 1.Single-variant families such as counterfactual_identification are unchanged along the output-variant axis.
A.9 Scoring rules by rung … A.16.2 Benchmark task prompt templates
The appendix defines rung-specific scoring, abstention routing, and aggregate leaderboard metrics, then analyzes model performance, restart variability, verbalization robustness, observation-layer hardness, and trajectory failures. It also specifies prediction-task prompt formatting, including CSV outputs and optional central 90% intervals.
- A.9 Scoring rules by rung: Non-identifiable targets or null-equivalent answers enter a uniform abstention pool; otherwise, outputs receive task-specific metrics including RMSE, Brier score, ROC-AUC, F1, interval score, exact match, or absolute error.Null-equivalent labels include none, null, unknown, missing field, and empty string.
- A.9 Scoring rules by rung: 33 active task variants feed three leaderboard metrics: Pass Rate pools 18 exact-match variants plus abstention, Med. NRel. Err pools 11 continuous variants, and Med. F1-Loss covers the remaining four.Interval scores use normalized interval scores with capped-mean aggregation, preserving propriety while limiting catastrophic-miss dominance.
- A.10 Realistic-exam breakdowns: Rung-2 content reaches 100% Cont. PR for every model, so its 28.6–92.9% Pass-Rate spread reflects abstention and identifiability judgments rather than execution of identified analyses.Rung 1 is saturated for five of six models, while its prediction-error band remains 0.276–0.361.
- A.10 Realistic-exam breakdowns: GPT-5.5 leads symbolic reasoning at 90.9% Pass Rate, followed by Claude Opus 4.8 at 86.4% and Gemini 3.1 Pro at 81.8%, while open models trail.Symbolically solvable families include causal_sketch, identification, counterfactual_identification, and forbidden-controls bias_diagnostic.
- A.11 Per-axis pass@k breakdowns: Restarting improves transient validity failures: Qwen 3.6 35B’s point coverage rises from 20/25 to 24/25 and interval coverage from 11/14 to 12/14 on a second attempt.Binary pass@k is nondecreasing, but all-restarts consistency remains 50.0–61.8% versus pass@3 of 76.5–91.2%.
- A.12 Robustness to the CauseNet-seeded verbalization: Verbalization sensitivity is concentrated in the weaker agent: Kimi is largely invariant with nontrivial spread on 2/10 bundles, whereas Qwen is materially more sensitive and can flip on non-identifiable counterfactuals.The bundle-8 lure resembles a two-proxy proximal-causal argument, but point identification requires completeness assumptions absent from the graph and unchecked by Qwen.
- A.13 Observation-layer hardness under matched scenes: Observation-layer hardness raises loss and failure rates: median survivor loss increases from 0.006 to 0.152 for Kimi and 0.005 to 0.163 for Qwen from clean to proxy_hard, while valid answers fall to 8/10 and 4/10.Frontdoor-type targets triggered abstentions despite being certified identifiable, because models relied on textbook backdoor reasoning.
- A.14 Per-trajectory failure modes on the scene presented in Fig. 2: On the agricultural instrumental-variable scene, agents reached the same over-claim through different trajectories: Kimi discarded a correct derivation, Gemma never questioned instrument identification, and GPT-5.5 abstained without exploration.This exposes failures that combine synthetic generation with first-class abstention grading.