PaperlayerView exact revision on arXiv

Source-linked AI summary

LLM-Augmented Causal Discovery: Probabilistic Fusion of Edge Existence and Orientation

Neville K. Kitson, Anthony Constantinou

arXiv:2608.27472v1cs.AI

TL;DR

BNSL from observational data can struggle with orientation, whereas LLM causal knowledge is broad but unreliable. The paper introduces PDGs to represent edge-state uncertainty and fuse both sources, improving F1 over the better solo source in 22 of 26 networks. The analysis attributes their gains to complementary skeleton recall and orientation accuracy.

  • Problem

    BNSL can leave causal orientations unidentified, while LLM-generated graphs may hallucinate edges, miss domain relationships, or reflect correlations rather than causal patterns.

  • Method

    The study represents edge existence and orientation uncertainty in PDGs and fuses BNSL and LLM graph outputs using weighted averaging.

  • Results

    Equal-weight fusion improves F1 over the better of BNSL and LLM alone in 22 of 26 networks, with a mean improvement of 0.056 (p < 0.001).

  • Takeaways & Limitations

    BNSL supplies higher-recall edge skeletons, while LLMs supply more accurate orientations, making their probabilistic combination practically effective.

  • Takeaways & Limitations

    The current PDG representation assumes causal sufficiency and lacks an explicit bidirected-edge state.

Abstract

from arXiv · show

Bayesian network structure learning (BNSL) from observational data struggles with orientation identifiability, while large language models (LLMs) offer broad but often unreliable causal knowledge. We propose combining these complementary sources through a novel representation, termed Probabilistic Dependency Graphs (PDGs). In a PDG, each edge is associated with a distribution over directed, undirected, and absent states, enabling fusion via weighted averaging. We evaluate this approach on 26 benchmark networks, combining ensembles of three BNSL algorithms (FGES, Tabu, PC) with three LLMs (Gemini, Claude, GPT) across multiple prompts and random seeds. A simple 50/50 fusion improves F1 over the better of either source alone in 22 of 26 networks, with a statistically significant mean improvement of $0.056$ $(p<0.001)$. Analysis reveals that the two sources play complementary roles: BNSL contributes a high-recall edge skeleton (80\% vs 60\% for LLM), while LLM contributes accurate edge orientation (96\% vs 77\% for BNSL). Our results show that representing both sources as probabilistic uncertainty over edge existence and orientation is a practical and effective way to improve causal graph accuracy.

1. Introduction

BNSL can leave causal orientations unidentified, while LLM-derived causal knowledge is broad but imprecise. The paper introduces PDGs to represent and fuse uncertainty from both sources.

  • BNSL may leave edges undirected or orient arcs arbitrarily, while assumptions about noise, latent variables, distributions, and functional relationships can reduce accuracy.
  • LLM-proposed graphs can hallucinate edges, miss domain-specific relationships, or encode correlations rather than causal patterns.
  • The study tests whether complementary BNSL and LLM errors can be combined to improve causal discovery accuracy.
  • Probabilistic Dependency Graphs represent edge existence and orientation uncertainty directly at the edge level and support weighted fusion.
  • The paper evaluates PDGs by introducing their representation and fusion procedure, then testing them on benchmark networks and analysing source complementarity.

2. Methods

The method maps BNSL and LLM outputs into probabilistic edge-state representations, fuses them, and greedily extracts a DAG for evaluation. Experiments use diverse ensembles across 26 discrete benchmark networks.

  • 2.1. Probabilistic Dependency Graphs: PDGs assign each canonical node pair probabilities for absent, forward, backward, and undirected edge states.The four probabilities sum to 1.0 and can represent DAG and CPDAG edges.
  • 2.1. Probabilistic Dependency Graphs: MergeGraphs converts DAGs, CPDAGs, PDAGs, and PDGs into a common representation before weighted averaging across graph sources.PDG values represent empirical support for edge states rather than calibrated posterior probabilities.
  • 2.1. Probabilistic Dependency Graphs: ToDagGreedy retains edges whose existence probability exceeds τ = 0.3, selects the most likely direction, and skips additions that create cycles.Edges are processed in decreasing order of existence probability, making the output a DAG through greedy inclusion.
  • 2.2. Generating the LLM Ensemble PDG: The LLM workflow uses neutral variable aliases, domain context, structured edge probabilities, two prompt detail levels, three models, and repeated high-temperature requests.Neutral aliases are used to mitigate benchmark memorisation risk, while ensemble diversity contributes to final quality.
  • 2.3. Generating the BNSL Ensemble PDG and Final Fused PDG: The BNSL ensemble combines PC-Stable, Tabu-Stable, and FGES, using sets of five learned graphs from different data samples to represent structural uncertainty.The primary experiments use 10,000-row samples, with 1,000-row results included in ablations.
  • 2.4. Evaluation: The evaluation covers 26 discrete networks with 8–70 variables and compares output DAGs with reference DAGs using F1.The networks span multiple application domains and come from three network repositories.

3. Results

Across 26 benchmark networks, fusion improves structural accuracy over either source alone, with gains linked primarily to correcting orientation errors. BNSL supplies higher edge recall, while LLMs supply more accurate directions, and diversity generally strengthens ensembles except when maximizing sample size is more important.

  • Structural accuracy: Fusion exceeds the better standalone source in 22/26 networks, with a mean F1 improvement of 0.056 (p < 0.001).The fused DAG also improves over BNSL in 23/26 networks and over LLM in 25/26.
  • Network size: Fusion gains are most consistent for networks with 15–40 variables, while gains for the largest networks are more modest.For very small networks, individual improvements can be large but less consistent.
  • Complementary roles: BNSL achieves higher edge recall than LLMs, detecting 0.802 of reference arcs versus 0.603 for LLMs.Across all networks, BNSL exclusively finds 445 reference arcs compared with 103 found exclusively by LLMs.
  • Complementary roles: LLMs achieve 0.96 orientation accuracy versus 0.77 for BNSL, and fusion corrects 67% of arcs with incorrect or weak BNSL orientations.Orientation-accuracy gains significantly correlate with overall F1 improvement (Spearman r = 0.51, p = 0.008).
  • Mechanism: The significant predictor of fusion improvement is orientation gain, whereas edge-existence complementarity and probability-weighted divergence show no significant relationship with F1 improvement.Orientation gain correlates with F1 improvement at r = 0.51 (p = 0.008), while existence metrics are nonsignificant.
  • Error analysis: LLM false positives usually reflect mediated effects or common causes rather than reversed true causal edges.Among LLM false positives, 31% are mediated effects, 30% involve common causes, and 7% are reversed true causal edges.
  • Ablations: Ensembling diverse models, prompts, repetitions, and BNSL algorithms improves performance, but 10K-only data outperform the 1K-plus-10K ensemble.The 1K-plus-10K ensemble reaches F1 = 0.729, versus F1 = 0.760 for 10K alone, producing a negative diversity gain.
  • Weighting: Oracle per-network weighting improves F1 by 0.027 over 50/50 fusion and by 0.083 over the better standalone source, but requires the reference graph.Because oracle weights are impractical without known references, equal weighting is presented as a reasonable robust heuristic.

4. Conclusion

The study combines BNSL and LLM-derived causal knowledge through PDGs, improving causal graph accuracy while retaining each source’s complementary strengths. The conclusion also identifies scope, calibration, weighting, validity, and real-world-data limitations.

  • Conclusion: 50/50 fusion improves F1 over the better solo source in 22 of 26 cases, with a mean gain of 0.056 (p < 0.001).The result uses a Wilcoxon signed-rank test.
  • Conclusion: The method requires no changes to underlying structure-learning algorithms and uses standard LLM APIs and models.PDGs merge source distributions over edge states using weighted averaging.
  • Conclusion: BNSL detects 80% of reference arcs versus 60% for LLMs, while LLMs achieve 96% orientation accuracy versus 77% for BNSL.The sources therefore contribute complementary edge-skeleton recall and orientation accuracy.
  • Conclusion: Orientation-accuracy gains significantly predict F1 improvement (r = 0.51, p = 0.008), identifying orientation correction as the primary mechanism of LLM benefit.Ablations also find positive contributions from diversity across LLM models, prompts, repeated requests, and BNSL algorithms.
  • Limitations: Evaluation is limited to clean synthetic data from fully observed discrete networks of up to 70 nodes, so real-world performance may be overestimated.Larger or continuous-variable networks, measurement noise, missing data, latent confounding, and model misspecification remain untested.
  • Limitations: PDG values are empirical support for edge states rather than calibrated posterior probabilities, and oracle weighting yields a mean additional F1 gain of 0.027.The study also reports unsuccessful early attempts to select weighting using edge variability or final-DAG BIC.

Appendix B. Networks used in the evaluation

The evaluation uses 26 discrete-variable benchmark networks with publicly accessible reference Bayesian networks selected from three repositories. Their reference structures and probability tables generate the synthetic data used for BNSL evaluation, while variable names are replaced with aliases in LLM requests.

  • Networks used in the evaluation: 26 discrete-variable networks were selected because each has a defined reference BN in a public repository.The repositories are bnlearn, Bayesys, and bnRep; reference structures were defined by expert elicitation or literature review.
  • Networks used in the evaluation: Reference DAGs and CPTs generate the synthetic datasets supplied to the BNSL algorithms.Twelve corical, two barley, and six water CPTs were adjusted to prevent degenerate single-valued variables.
  • Networks used in the evaluation: Canonical network names are withheld from LLMs, which receive variable-name aliases instead.This naming choice supports evaluation without presenting the benchmark’s canonical identifiers in LLM requests.

Appendix C. Detailed per-network Fusion Results

Appendix C reports per-network fusion results across F1, DAG SHD, CPDAG SHD, and optimal weighting. Fusion is computed by first forming within-source consensus PDGs and then combining BNSL and LLM sources with equal weighting.

  • Detailed per-network Fusion Results: Fusion compares the better of BNSL and LLM against a two-stage 50/50 merge of their consensus PDGs.This design gives each source equal contribution regardless of how many constituent PDGs it contains.
  • DAG SHD: DAG SHD decreases by a mean of 1.7, improving in 15 of 26 networks, but the gain is less pronounced than for F1.The paper attributes this metric difference to F1 penalizing misorientation more strongly than SHD.
  • CPDAG SHD: CPDAG SHD is better than both solo sources in 11 of 26 networks, while fusion slightly worsens overall CPDAG SHD against the better solo result.CPDAG metrics reward preservation of equivalence-class uncertainty and therefore favor structure-learning outputs.

Appendix D. BNSL Ensemble Effect: Per-network Detail

The appendix examines whether aggregating runs from three BNSL algorithms improves performance over averaging individual BNSL runs. It reports the comparison per network using 10K-sample datasets and F1.

  • BNSL Ensemble Effect: Table 9 reports the ensemble effect separately for each network using 10K-sample datasets.The appendix compares mean individual-algorithm-run F1 with ensemble F1.
  • BNSL Ensemble Effect: The BNSL ensemble is formed by merging individual FGES, Tabu-Stable, and PC-Stable runs into an ensemble PDG and deriving an ensemble DAG.The comparison uses mean F1 across individual runs versus the merged three-algorithm ensemble.

E.1. Motivation

The audit examines whether benchmark recognition or context-generation leakage could explain some of the LLM performance observed in the study. It investigates both mechanisms through a post-hoc review of the network context files.

  • E.1. Motivation: The audit addresses concern that LLM performance might reflect benchmark recognition or information leakage rather than causal reasoning.The two investigated mechanisms are benchmark recognition and context-generation leakage.
  • E.1. Motivation: A post-hoc audit was conducted on all network context files used in the experiments.

E.2. Audit Methodology

The audit used Claude Opus 4.5 to qualitatively assess benchmark identifiability and causal information revealed by the supplied contexts. It compared minimal and standard prompts under obfuscated and original variable names.

  • E.2. Audit Methodology: Claude Opus 4.5 assessed the likelihood that each benchmark could be identified from its supplied context.The auditor also justified scores by identifying contributing information.
  • E.2. Audit Methodology: Claude Opus 4.5 assessed the extent to which supplied context information revealed causal structure.The qualitative score scales are shown in Table 10.
  • E.2. Audit Methodology: The audit’s score scales are documented in Table 10, while benchmark-identification scores are reported in Table 11.
  • E.2. Audit Methodology: The audit used four settings combining minimal or standard prompts with obfuscated or original benchmark variable names.

E.3. Audit Results

Benchmark identifiability varied substantially across networks, while estimated causal-information leakage was generally mild and changed little across prompt levels and naming regimes. The audit also identified concrete semantic clues that could reveal benchmark identity or causal roles.

  • E.3. Audit Results: Several well-known networks remained highly identifiable even with obfuscated variable names, whereas many lesser-known networks were more resistant.Claude attributed identification to variable counts, variable types, and variable names; the alarm network was identified using 37 variables and heart-rate-related groups.
  • E.3. Audit Results: Estimated causal-information leakage was relatively mild for many networks and generally similar across minimal and standard prompts and naming regimes.The highest estimated leakage values occurred for corical and vessel1.
  • E.3. Audit Results: Examples of causal clues included disease-related variable names, explicit relationships, and words such as “outcome,” “score,” “baseline,” and “input.”These terms were judged to suggest leaf or upstream causal roles in some contexts.

E.4. Summary and Future Work

The audit provides evidence against benchmark recognition or context-generation leakage being the primary explanation for fusion gains, but it cannot conclusively resolve their contribution. It recommends stronger controls and more novel benchmarks for future work.

  • E.4. Summary and Future Work: The audit cannot conclusively determine whether benchmark memorisation or causal-information leakage contributed to the observed results.Its scores are informed qualitative judgements generated by an LLM rather than objective measurements.
  • E.4. Summary and Future Work: Variable-name obfuscation substantially reduced benchmark identifiability for many networks.
  • E.4. Summary and Future Work: Benchmark-identification scores and estimated causal-information leakage were not associated with fusion improvement.
  • E.4. Summary and Future Work: Future audits could remove variables, eliminate causal clues from aliases, and use lesser-known or theoretically grounded novel benchmark networks.
Loading 2608.27472v1…