Externalized CPDAG Summaries Improve LLM Causal Deduction

1 Introduction

• A plausible causal story is insufficient: the claim must hold in every compatible graph, whose family is represented by a CPDAG with forced directions and ambiguous edges.Corr2Cause supplies marginal dependencies and conditional independencies, then asks whether a natural-language causal hypothesis is deducible.Source: §1 · pp. 1–2
• Free-form reasoning can collapse the all-compatible-graphs question into local pattern matching, despite the need to track adjacency, separating sets, colliders, and edge ambiguity.The paper frames this as latent-object reasoning: the label is defined by a CPDAG query.Source: §1 · p. 2
• Structured Thinking emits a typed CPDAG summary before answering, with fields for variables, edges, separating sets, v-structures, and orientations.The emitted graph is returned to the model for the final causal-hypothesis query; the method uses neither fine-tuning nor a symbolic causal solver.Source: §1 · p. 2
• The primary comparison uses a strong PC-instruction baseline, while a two-turn prose control tests whether the same PC scaffold works without a typed graph intermediate.The direct baseline explicitly removes edges, orients v-structures, propagates orientations, and requires YES only when the claim holds in every compatible DAG.Source: §1 · p. 2
• On the full Corr2Cause test, Qwen3.5-27B reaches 86.4 F1(YES) versus 73.0 for the PC-instruction baseline, a +13.4 pp gain in the primary run.Across three full-test seeds, the mean gain is +8.1 ± 5.3 pp; the PC-scaffolded prose control reaches 67.6 F1.Source: §1 · p. 2
• The introduction reports evidence across open-weight and API models, Paraphrase-OOD evaluation, ablations, graph-content probes, and a full-split CPDAG-field audit.The audit finds the emitted graph state usually close to the reference object and predictive of downstream correctness.Source: §1 · p. 2

CPDAGs and causal deduction

• A CPDAG represents the Markov-equivalence class: directed edges are forced across compatible graphs, while undirected edges retain ambiguity.A plausible DAG can still support a wrong answer if another DAG in the equivalence class falsifies the hypothesis.Source: §2 · p. 3

LLMs and structural causality

• Structural causal reasoning remains harder for LLMs than commonsense causal association, and prompting still leaves a gap to algorithmic ground truth, especially as variables increase.Corr2Cause labels follow independence constraints and PC-algorithm semantics, unlike benchmarks focused on natural-language explanation or SCM-world reasoning.Source: §2 · p. 3

Structured intermediates

• Structured Thinking makes the intermediate a constrained CPDAG summary, rather than a knowledge-graph-style intermediate or prose sub-answers.Explicit graph fields can be removed, scrambled, or queried; prior structured-intermediate work for Corr2Cause is attributed to Sun et al. [Sun et al., 2025].Source: §2 · p. 3

Faithfulness of reasoning traces

• The graph interventions test dependence on emitted-state content behaviorally, not full mechanistic faithfulness of every token.The paper distinguishes this scope from activation-level probes and broader process-faithfulness methods.Source: §2 · p. 3

Tool use, programs, and positioning

• The paper uses a tool interface to record schema-constrained analysis, not to call a symbolic causal solver; its claim is confined to CPDAG-query tasks.The stated comparison is against a strong PC-scaffolded direct baseline and a PC-scaffolded two-turn prose control.Source: §2 · p. 3

3.1 Task

• Under faithfulness and Markov assumptions, premises identify a CPDAG, and a positive label requires the hypothesis to hold in every DAG in its equivalence class.Source: §3.1 · pp. 3–4

3.2 Structured Thinking pipeline

• Structured Thinking emits a schema-constrained CausalAnalysis graph in Turn 1, returns it with a compact DOT view, and answers the hypothesis against that state in Turn 2.The materialized graph separates adjacency, compelled colliders, and forced orientations; six-variable instances use short lists rather than long derivations. The schema ensures parseable fields, not full CPDAG consistency.Source: §3.2 · p. 4

3.3 Baselines and controls

• The controls compare explicit PC guidance in one-turn direct answers and two-turn prose against typed graph externalization.A single-turn structured-prompt diagnostic tests whether graph vocabulary alone can replace explicit graph-object externalization.Source: §3.3 · p. 5

4.1 Setup

• The ID test has n=1162, while Paraphrase-OOD has n=2246 and preserves graph instances while changing wording.Qwen-family structured runs use schema-derived regex constraints; direct baselines do not emit a constrained CPDAG object.Source: §4.1 · p. 5

4.2 Cross-model results

• Across three full-ID seeds, Structured Thinking averages 85.57 ± 1.70 F1 versus 77.49 ± 4.07 for BASELINEENHANCED, a gain of +8.08 ± 5.34 pp.The additional-seed error shifts trade small precision decreases for larger recall improvements.Source: §4.2 · p. 5
• The PC-scaffolded prose control, TWOTURN-PROSE-PC, reaches 67.55 F1, below both the one-turn PC-instruction baseline and Structured Thinking, while its higher recall comes with many more false positives.It has 71 false positives versus 27 for the one-turn baseline.Source: §4.2 · p. 5
• F1(YES) rises from 73.01 to 86.36 (+13.35 pp) for Qwen3.5-27B on the full ID test versus the one-turn PC-instruction baseline.The primary paired comparison has a bootstrap 95% CI of [+8.42, +18.59] and exact McNemar p = 2.4 × 10−6.Source: §4.2 · p. 5, Table 1 · p. 6
• Gains also appear on Qwen3.6-27B (+9.79 pp), Paraphrase-OOD (+10.46 pp), and GPT-5.4-mini (+3.19 pp).For Qwen3.6-27B, a single-turn structured-prompt diagnostic reaches 74.53 F1, below its 75.92 baseline.Source: §4.2 · p. 6

4.3 Effect scales with task complexity

• Pilot larger-graph tests improve Qwen3.5-27B F1 on Random n=7 (28.42 to 91.87), ALARM-derived n=7 (15.38 to 88.73), and ASIA-derived n=8 (9.30 to 60.00).These are not a controlled variable-count sweep; Qwen3.5-9B also improves but has much lower absolute F1, suggesting capacity sensitivity.Source: §4.3 · p. 6
• On the hardest informative bins, gains widen from +5.3 pp at n=5 to +18.6 pp at n=6 on ID, and from +5.9 to +13.6 pp on Paraphrase-OOD.Bins at n≤4 are omitted because they have too few positives for informative F1 estimates.Source: §4.3 · p. 6, Figure 2 · p. 7

• These one-factor removals are sensitivity tests, not additive effect estimates, because they are not a full factorial decomposition.Source: §5 · p. 7
• Replacing the PC-algorithm prompt with a minimal causal-analysis instruction drops F1 to 36.2, far below baseline.The authors interpret this as evidence that causal-discovery content matters, rather than any tool call being sufficient.Source: §5 · p. 7
• Removing thinking costs 14.4 pp and removing constrained decoding costs 9.5 pp at 27B.The constrained-decoding ablation retains tool-call syntax but removes the hard regular-language constraint.Source: §5 · p. 7
• At 9B, thinking becomes harmful while constrained decoding remains helpful, suggesting reasoning traces should be calibrated by model size.Source: §5 · p. 7

• Scrambling the full CPDAG drops F1 by 12.0 pp, compared with 3.8 pp when the Turn-2 premise is removed.The full-split Qwen3.5-27B ID probe tests whether the model uses the graph rather than merely rereading the premise.Source: §6 · pp. 7–8, Table 3 · p. 8
• Skeleton, v-structure, and directed-edge scrambles each lower F1 by 5.7–6.4 pp, countering both a pure adjacency-extraction account and a shallow answer-key account.Source: §6 · p. 8
• These destructive probes demonstrate behavioral dependence on graph content, not a formally valid MEC proof on every instance.A scrambled graph may no longer be a valid CPDAG.Source: §6 · p. 8

7.1 Error localization and error shifts

• In the primary seed, positive-class misses fall from 61 to 28 and false positives from 27 to 20; a 32-case diagnostic also finds errors in graph state, Turn-2 readout, or MEC ambiguity handling.Source: §7.1 · pp. 8–9
• On ID, emitted fields score 0.960 skeleton F1, 0.923 v-structure F1, and 0.896 directed-edge F1, with 75.9% exact CPDAG match.Paraphrase-OOD values are similar: 0.959, 0.920, 0.884, and 74.4%, respectively.Source: §7.1 · p. 8
• Graph-consistency violations occur in 18.4% of ID outputs and 20.7% of Paraphrase-OOD outputs; accuracy declines as graph quality falls on both splits.On ID, accuracy is 97.2% for exact matches, 93.9% for valid non-exact graphs, and 91.1% for summaries with consistency violations.Source: §7.1 · p. 8

7.2 Ruling out simpler explanations

• The PC-scaffolded two-turn prose control reaches 67.55 F1, below the one-turn PC baseline (73.01) and Structured Thinking (86.36), so the prose interface alone does not explain the gain.A single-turn CPDAG-oriented prompt without an external graph object also scores below the standard Qwen3.6 baseline.Source: §7.2 · p. 9

7.3 Cost, reproducibility, and deployment

• The method uses two model calls and roughly twice the tokens and 1.5× wall-clock time of the one-turn baseline, without fine-tuning or a symbolic causal solver.The authors recommend validating the PC prompt, constrained decoding, and thinking traces at the target scale.Source: §7.3 · p. 9

8 Limitations

• The results are scoped to CPDAG-style deduction from conditional-independence statements; transfer to other query settings is not established.Other settings would require a changed intermediate representation.Source: §8 · pp. 9–10
• Several evaluations remain single-seed, so smaller gains on already strong baselines should be read directionally.This applies to Paraphrase-OOD, larger-graph stress tests, cross-model rows, and several controls.Source: §8 · p. 9
• Schema validity does not certify acyclicity, Meek closure, or full internal consistency; graph-consistency violations occur in 18–21% of outputs.Certified CPDAG validators remain useful future work.Source: §8 · p. 9
• The controls do not isolate every component because schema constraints, graph-field names, tool-result formatting, and Turn-2 instructions remain bundled.Lightly structured prose and validity-preserving graph interventions could sharpen the mechanism claim.Source: §8 · pp. 9–10

9 Conclusion

• +13.35 pp F1(YES) on Qwen3.5-27B ID in the primary paired run, with positive gains across two additional full-ID seeds and other reported settings.The conclusion describes the PC-scaffolded prose control and single-turn structured-prompt diagnostic as failing to recover the gain.Source: §9 · p. 10

A Additional diagnostics

• False negatives fall from 61 to 28 and false positives from 27 to 20 for Structured Thinking on the primary Qwen3.5-27B ID seed.The PC-scaffolded prose control increases recall but introduces many false positives.Source: Table 5 · p. 12

A.1 API configuration sensitivity

• With GPT-5.4-mini, tool_choice="auto" plus a prompt instruction restores reasoning budget and yields 88.6 F1, unlike name-forcing the tool.Name-forcing reduced F1 to 74.7 versus the 85.4 baseline; the configurations differ in API flag and prompt instruction.Source: §A.1 · p. 12

B Full-ID seed robustness

• Across the full-ID runs, Structured Thinking improves F1 and accuracy every time, though the gain varies and not every error type improves on every seed.The primary seed improves precision and recall; the additional seeds are mainly recall-driven with small precision costs.Source: B Full-ID seed robustness · p. 12

C Larger-graph stress tests

• The larger-graph stress tests show large 27B gains and positive but capacity-limited 9B rows; the ASIA-derived 9B row has N=40 and is directional.These single-seed tests use graph families outside Corr2Cause’s original n≤6 range and do not replace the paired full-ID comparison.Source: C Larger-graph stress tests · p. 13, Table 7 · p. 13

D Scale interaction diagnostic

• At 9B, disabling thinking improves Structured Thinking by +4.9 pp and the baseline by +6.4 pp; constrained decoding helps more at 9B than 27B.The reported constrained-decoding gains are 13.7 pp at 9B and 9.5 pp at 27B; the authors suggest enabling thinking only when validation shows reliable traces at the target scale.Source: D Scale interaction diagnostic · p. 13, Table 8 · p. 13

E.1 Structured Thinking system prompt (Sstruct, Turn 1)

• The Structured Thinking prompt walks through skeleton construction, collider orientation, Meek rules, and recording the CPDAG before the tool call.The prompt uses a closed-world assumption for stated independencies.Source: §E.1 · p. 14

E.2 Turn-2 tool-result template

• Turn 2 checks the queried edge and orientation, then tests undetermined edges across all valid DAG completions.The prompt also directs systematic case analysis for indirect-causation claims.Source: §E.2 · pp. 14–15

E.3 BaselineEnhanced system prompt (Sbase)

• The direct-answer baseline supplies a PC-algorithm scaffold and requires a causal hypothesis to hold in every DAG in the Markov equivalence class.The comparison controls for explicit PC content, not token-identical prompts or externalization alone.Source: §E.3 · p. 15

E.4 Control prompt variants

• The prose control uses the same PC scaffold across two turns but withholds a typed graph object and tool call.It tests whether a prose intermediate with that scaffold is sufficient.Source: §E.4 · pp. 15–16

F.1 CausalAnalysis schema

• Turn 1 records variables, skeleton, non-adjacent pairs with separating sets, v-structures, directed edges, and undirected edges in a CausalAnalysis object.The schema specifies these graph fields but does not itself enforce full CPDAG consistency.Source: §F.1 · p. 16

F.2 Regex construction (Qwen / Mistral)

• Qwen-family decoding uses a schema-derived regex, while API models enforce the same schema through tool calling.The main GPT-5.4-mini run uses tool_choice="auto" with a prompt instruction to call the tool.Source: §F.2 · pp. 16–17

G Per-experiment tables

• The appendix provides per-experiment evaluation and usage files, prompts, schemas, scripts, and assets; no new model weights or human-subject data are released.The failure gallery hand-labels 32 representative errors and is diagnostic, not an unbiased prevalence estimate.Source: G Per-experiment tables · p. 17, H Assets and licenses · p. 17, I Failure-case gallery · p. 17

J Reproducibility

• A single command using the released code reproduces every experiment.The Corr2Cause dataset is used as released.Source: J Reproducibility · p. 18
• Run scripts cover Qwen3.5-9B main and ablation runs, full-split probes, additional diagnostics, and Paraphrase-OOD.Appendix B records Qwen3.5-27B full-ID seed reruns in the same per-experiment metadata format.Source: J Reproducibility · p. 18
• Metadata files record decoding arguments, model and tokenizer revisions, API model snapshots, and package versions for each experiment.Open-weight runs use 2× GPU tensor-parallel vLLM inference with gpu_memory_utilization=0.88; API runs use max_concurrency=5.Source: J Reproducibility · p. 18