PaperlayerView exact revision on arXiv

Source-linked AI summary

Active Causal Discovery Benchmark: Evaluating LLM Agents Under Budgeted Interventions

Sagar Deb, Devam Shah, Ashwanth Krishnan

arXiv:2609.31675v1cs.LGcs.AI

TL;DR

It remains unclear whether LLM interaction with data and budgeted interventions yields reliable causal structure recovery rather than plausible-looking graphs. ACDB provides a shared benchmark and scoring contract; PC with greedy active orientation is strongest on the current ladder, while the results remain a calibration audit rather than evidence that LLMs solve the task.

  • Problem

    It remains unclear whether LLM agents can reliably recover causal structure through observation and budgeted interventions, despite their ability to inspect data, choose tests, intervene, and submit graphs.

    Source: §1 · p. 1

  • Method

    ACDB evaluates agents on paired linear–Gaussian SCM instances through a fixed observe–intervene–submit API and a shared scoring contract.

    Source: §1 · p. 1, §3 · p. 3

  • Results

    PC + greedy leads on the v0 ladder (42.7% directed F1, SHD 4.79), versus the best LLM at 31.7% directed F1 (SHD 7.25) and random at 23.6% directed F1.

    Source: §8 · p. 11

  • Takeaways & Limitations

    ACDB is a scaffold for tracking active causal discovery as models and benchmark calibration improve.

    Source: §8 · p. 11

    LLM capability conclusions are conditional on the two tested backbones and one prompt design per backbone.

    Source: §6.4 · p. 10

Abstract

from arXiv · show

We introduce the Active Causal Discovery Benchmark (ACDB), an SCM-grounded environment for evaluating whether LLM agents recover causal graph structure from observations and budget-constrained hard interventions. ACDB pairs a linear-Gaussian world generator with a fixed observe-intervene-submit API and a three-layer scoring contract that separates skeleton recovery, DAG recovery, and intervention efficiency. On the current six-level ladder, PC with a greedy active orientation heuristic is the strongest non-oracle method (directed F1 42.7%, SHD 4.79), ahead of Claude Sonnet 4.6 raw active (31.7%, 7.25) and GPT-5.4 raw active (22.9%, 9.27). The most informative diagnostic is the precision-recall decomposition: PC under-commits with high precision, LLMs over-commit with lower precision, and statistical-tool access often increases abstention rather than useful intervention. A structure-blind random DAG baseline reaches 23.6% directed F1 on this dense v0 ladder; a density probe lowers this floor to 16.9%, motivating the v1 calibration pass. The current results should therefore be read as a benchmark audit and calibration report, not as evidence that current LLMs solve active causal discovery.

1 Introduction

The paper asks whether LLM agents can use budgeted hard interventions to recover causal DAGs better than a structure-blind baseline. It introduces ACDB to audit that question and reports distinct error patterns for PC and LLMs.

1 Introduction

  • Classical methods can be brittle at small samples and do not specify how an agent should allocate a finite intervention budget, motivating the active-agent question.The paper asks whether LLMs can outperform a structure-blind random baseline, and by how much and under what conditions.Source: §1 · p. 1
  • ACDB pairs a linear–Gaussian SCM generator and fixed agent protocol with a shared scoring contract and six-level difficulty ladder.The contribution is a benchmark and controlled empirical audit, not a new causal discovery algorithm; its scope assumes causal sufficiency, linear–Gaussian SCMs, and perfect hard interventions.Source: §1 · pp. 1–2
  • PC with greedy active orientation is strongest among non-oracle methods (directed F1 42.7%, SHD 4.79); PC under-commits while LLMs over-commit, and random DAGs reach 23.6% directed F1.PC has high precision (66.9%) and low recall (32.8%); LLMs have lower precision and comparable recall.Source: §1 · p. 2
  • Statistical-primitive tools did not improve directed F1 and often shifted models toward more testing and less intervention, yielding high efficiency through abstention.At L2 and L5, Sonnet 4.6’s observational directed F1 exceeded its active-panel score on the same seeds.Source: §1 · p. 2

2 Background

The background establishes what observations can identify, which assumptions underpin the benchmark, and why active discovery must score observational recovery separately from intervention-based orientation.

2 Background

  • Observations identify a CPDAG, which fixes the skeleton and compelled directions but leaves some edges undirected; interventions or structural assumptions are needed to select a DAG within the class.Markov-equivalent DAGs share a skeleton and v-structures [Verma and Pearl, 1990, Andersson et al., 1997].Source: §2 · p. 2
  • ACDB assumes faithfulness, causal sufficiency, and linear–Gaussian noise, excluding latent confounders and nonlinear mechanisms to make scoring identifiable and PC a meaningful baseline.The generator rejects cases approaching the faithfulness boundary because violations can be troublesome at finite samples.Source: §2 · pp. 2–3
  • PC tests conditional independences to remove adjacencies and orients edges using v-structures and Meek’s rules; under stated assumptions and a correct oracle, it returns the true CPDAG.With finite samples, its output is an empirical approximation dependent on the independence test and significance level.Source: §2 · p. 3
  • Active discovery combines observational recovery of the skeleton and compelled edges with intervention-based resolution of remaining undirected edges, which depends on policy quality.The benchmark scores these two jobs separately.Source: §2 · p. 3

3 The ACDB Benchmark

ACDB provides a controlled environment for agents to observe data, use a budget of hard interventions, and submit causal graphs. Its scoring and difficulty ladder distinguish recovery quality, intervention use, and benchmark calibration.

3 The ACDB Benchmark

  • ACDB hides the ground-truth DAG and SCM while exposing observational data, variable count, and intervention budget; agents spend hard single-node interventions before submitting mixed directed and undirected graphs.The fixed protocol allows unresolved edges and scores them separately from reversals or omissions.Source: §3 · p. 3, §3.2 · p. 3

3.1 World generation

  • Accepted instances are sampled, converted to CPDAGs, screened for invalid or near-unfaithful cases, parameterized as linear–Gaussian SCMs, assigned optimal intervention sets, anonymized, and sampled for data.Rejection gates return the process to graph sampling.Source: Figure 2 · p. 4

3.3 Scoring contract

  • The contract separates skeleton and compelled-edge F1, directed F1 and SHD, and intervention efficiency; the efficiency score is one at or below the optimal intervention count and falls with wasted interventions.The DAG-layer SHD counts extra, missing, reversed, and unresolved edges; it does not provide MEC-aware partial credit.Source: §3.3 · pp. 3–5, Figure 3 · p. 4

3.4 The difficulty ladder

  • The six-level ladder varies graph size and density, statistical power, and budget tightness, with L2 and L4 designed to isolate statistical and budget axes while holding graph size fixed.Levels are evaluated on eight paired seeds using the same manifest across agents and baselines.Source: §3.4 · p. 5, Table 1 · p. 5

3.5 Density leakage: a benchmark-design hazard

  • 23.6% to 16.9% overall random directed-F1 floor under the density-probe ladder (ρ ≤ 0.33 on non-tutorial levels); Equation (3) is a probe fit, not a derived bound.On dense graphs, a random DAG can match a non-trivial fraction of directed edges, creating density leakage.Source: §3.5 · p. 5

4 Agents and Baselines

ACDB compares LLM policies with PC-based, random, and correlation-only baselines under a shared session API. The configurations vary in data access, statistical tools, and use of interventions.

Shared evaluation

  • All agents see the same preflighted seeds and scoring contract, differing only in their session policy.The shared manifest contains 8 seeds per level.Source: §4 · p. 5

Structure-blind random baseline

  • 23.6% to 16.9%: lowering density reduces the structure-blind random baseline’s overall directed-F1 floor.The baseline samples using only the public variable count, without observational data, edge count, or interventions.Source: Figure 4 · p. 6, §4 · p. 6

PC and PC + greedy active

  • 42.7% directed F1 and 4.79 SHD make PC with greedy active orientation the strongest non-oracle method described.PC starts from observational data, then intervenes on the highest undirected-edge-degree node and orients unresolved neighbors by mean shift.Source: §4 · pp. 5–6

LLM agents

  • LLM tests compare GPT-5.4 and Claude Sonnet 4.6 across raw or statistical-tool configurations and observational or active panels.The statistical variant adds correlation, partial-correlation, and Fisher-z tools; raw episodes have a 16-step limit and statistical episodes a 32-step limit.Source: §4 · p. 6

Correlation-only ablation probe

  • The correlation-only probe submits a graph from correlations and standard deviations, without raw data, tools, or interventions.It is a diagnostic ablation on levels 1, 3, and 5, using four seeds per level and both backbones.Source: §4 · p. 6

5 Results

On the six-level ladder, PC+greedy is the strongest non-oracle method, while LLMs show different precision–recall errors and do not reliably turn intervention budgets into orientation accuracy.

5 Results

  • The run planned 336 jobs per model across the ladder, with eight shared seeds per level; the results are a benchmark audit, not evidence that current LLMs solve active causal discovery.GPT-5.4 completed all 336 jobs and Sonnet 4.6 completed 335; paired comparisons therefore used identical instances.Source: §5 · pp. 6–7, §5.6 · p. 9

5.1 Headline results

  • PC+greedy had 66.9% directed precision and 32.8% recall, whereas Sonnet raw was 35.0%/29.6% and GPT-5.4 raw was 36.3%/19.3%.The decomposition shows PC under-committing with higher precision and LLMs over-committing with lower precision; F1 alone obscures this distinction.Source: §5.1 · p. 7
  • Statistical-tool access did not improve directed F1: Sonnet raw scored 31.7% versus 28.0% for stats, and GPT raw scored 22.9% versus 18.8%.Stats agents intervened less and submitted fewer directed edges, so their high efficiency reflected abstention rather than better active discovery.Source: §5.1 · p. 7
  • PC+greedy achieved directed F1 of 42.7% and SHD of 4.79, ahead of Claude Sonnet 4.6 active raw at 31.7% and 7.25.The paired comparison favored PC+greedy over the best LLM by 11.0 directed-F1 points and 2.46 SHD edges.Source: §5.1 · p. 7, Table 2 · p. 8

5.2 Per-level breakdown

  • At L2 and L5, Sonnet raw exceeded PC+greedy and its own active panel, while GPT-5.4 settings often approached the random floor.The Sonnet observational-versus-active gaps were not significant at eight seeds (p = 0.50, p = 0.20); low-sample levels also produced PC’s largest finite-sample errors.Source: §5.2 · p. 8, §5.3 · p. 9

5.3 Density leakage: the random floor

  • The structure-blind random DAG baseline reached 23.6% overall directed F1; a density probe lowered the measured floor to 16.9%, motivating v1 density recalibration.The v0 ladder remains the basis for existing LLM and PC runs.Source: §5.3 · p. 9

5.4 Correlation-only ablation probe

  • The correlation-only ablation had non-trivial skeleton F1 of 67–74% but directed F1 of 0 under a prompt requiring models to withhold unsupported orientations.It used standard deviations and a rounded correlation matrix on L1/L3/L5, with four seeds per level and both backbones; it is not a primary DAG-recovery comparison.Source: §5.4 · p. 9

5.5 Efficiency: budgets were rarely the binding constraint

  • LLM efficiency ranged from 92.7% to 100.0%, but high efficiency without precision/recall gains reflected abstention; PC+greedy reached 89.6%.Both stats settings were exactly 100.0% efficient and rarely intervened, while PC+greedy sometimes used redundant interventions.Source: §5.5 · p. 9

6 Discussion

The discussion finds that PC with greedy orientation remains strongest on the v0 ladder, while LLMs and PC exhibit distinct commitment errors. It identifies density leakage, scoring design, and limited evaluation scope as important to interpreting the results and planning v1.

6.1 What the results say about current LLMs

  • PC + greedy achieved the highest directed F1 and lowest SHD on the ladder, consistent with the v0 assumptions.The authors conclude that current LLM agents are not drop-in replacements for classical structure-learning algorithms within this identifiability regime.Source: §6.1 · p. 9
  • PC's precision 66.9 and recall 32.8 contrast with LLM precision 35–46 and recall 20–30, exposing under- versus over-commitment.Under-committers leave useful edges unresolved, while over-committers add wrong orientations; F1 alone collapses this distinction.Source: §6.1 · p. 9
  • At L2, 39.3% vs 36.9%, and L5, 40.1% vs 31.9%, Sonnet 4.6's observational directed F1 exceeded its active score on identical seeds.The authors interpret this as a policy diagnostic: current prompts do not reliably convert intervention samples into better orientation decisions.Source: §6.1 · p. 9
  • Statistical-tool variants did not improve directed F1 over raw data; their 100% efficiency reflected abstention rather than skill.These variants ran longer trajectories, intervened less, and submitted fewer directed edges.Source: §6.1 · p. 10

6.2 What the results say about the benchmark

  • The random baseline's 23.6% directed F1 falls to 16.9% with the density probe, making ladder calibration a concrete v1 target.The v0 comparisons remain useful because agents share paired seeds on the same ladder.Source: §6.2 · p. 10
  • ACDB's separate skeleton, compelled-edge, DAG, SHD, and efficiency scores distinguish CPDAG-ceiling, orientation, and policy-quality questions.For example, skeleton F1 alongside directed F1 makes a correlation-only probe's zero directed score interpretable as withheld orientation under the prompt-restricted reading.Source: §6.2 · p. 10
  • V1 should retain the SCM foundation while fixing density leakage, adding the full LLM-correlation summary condition, increasing seeds, and inspecting intervention-driven revisions.The paper also names causal inference with a known DAG, counterfactual queries, decision-making under causal uncertainty, transfer, and RL for experimental design as scaffold extensions, not v0 findings.Source: §6.3 · p. 10

6.4 Limitations

  • Interpretation is limited by causal sufficiency and linear–Gaussian assumptions, dense-end density leakage, eight seeds per level, and two LLM backbones with one prompt each.Eight seeds support coarse trends but are under-powered for fine-grained ranking; capability claims depend on the tested backbones and prompts.Source: §6.4 · p. 10

7 Related Work

ACDB draws on classical causal discovery, LLM causality research, and interactive-agent benchmarks. It differs by requiring interaction with data and interventions before a scored graph is submitted, within an inspectable formal SCM task.

Classical causal discovery

  • PC, Markov-equivalence theory, GES, and intervention-design work define the CPDAG ceiling and classical baselines for ACDB [Verma and Pearl, 1990, Andersson et al., 1997] [Eberhardt et al., 2005, Hauser and Bühlmann, 2012].Source: §7 · p. 10

LLMs for causality

  • Prior LLM causality work spans optimistic pairwise direction results and critiques of pattern matching, while CLadder and Long et al. study causal QA and text-only graph reconstruction [Kıcıman et al., 2023] [Zeˇcevi´c et al., 2023] [Jin et al., 2023] [Long et al., 2023].ACDB instead requires interaction with data and interventions before graph commitment.Source: §7 · p. 10

Agent benchmarks

8 Conclusion

The v0 ladder places PC + greedy above the best LLM and random baseline, while precision–recall behavior and statistical-tool abstention reveal distinct agent failure patterns. ACDB is presented as a scaffold for tracking progress, with calibration and revision analysis remaining next steps.

8 Conclusion

  • PC + greedy reached 42.7% directed F1 and SHD 4.79, versus the best LLM at 31.7% directed F1 and SHD 7.25 and random at 23.6% directed F1.The decomposition shows PC under-commits with high precision, LLMs over-commit with lower precision, and statistical-tool variants often abstain from intervention.Source: §8 · p. 11

Related papers

Loading 2609.31675v1…