PaperlayerView exact revision on arXiv

Source-linked AI summary

LLM Explainability with Counterfactual Chains and Causal Graphs

Nirit Nussbaum-Hoffer, Nitay Calderon, Liat Ein-Dor, Roi Reichart

arXiv:2606.05972v1cs.LG

TL;DR

LLM inference is difficult to explain because decision factors are opaque and existing approaches often lack faithful, global accounts of concept-level reasoning. The paper constructs causal graphs from text through LLM-discovered concepts and counterfactual augmentation, finding convergent expansion and stronger predictive value for discovered causal parents across tasks.

  • Problem

    Existing explainability methods do not provide an applicable global causal account of how an LLM organizes human-interpretable concepts into predictions.

  • Method

    The paper uses a four-phase pipeline to discover concepts, map concept states, generate counterfactual chains with MCMC-inspired augmentation, and construct causal graphs of LLM inference.

  • Results

    Across three LLMs and classification tasks, the expansion reaches distributional and topological convergence, while discovered causal parents predict nodes more strongly than alternative concept sets.

  • Takeaways & Limitations

    Concept-level causal graphs provide a framework for representing how LLMs organize information and produce predictions in a more accessible interpretability form.

  • Takeaways & Limitations

    Counterfactual generation relies on the target LLM for annotation, generation, and self-assessment, so errors in these steps may propagate.

Abstract

from arXiv · show

Causal graphs provide a high-level language for making mechanisms transparent. Recent work uses Large Language Models (LLMs) to recover causal graphs of external-world processes. Instead, in this paper, we use causal graphs to model LLM inference itself, providing stakeholders with a transparent view of how the model perceives and organizes high-level concepts to produce a prediction. We propose a four-phase method for constructing such graphs. Given a target LLM and a set of textual examples, our method discovers class-discriminative, human-interpretable concepts and maps each input to LLM-perceived concept states. We then introduce an MCMC-inspired counterfactual augmentation procedure that expands the sparse observational data through chains of counterfactuals. This enables stable causal discovery with $σ$-CG, yielding informative, interpretable graphs. We apply our method to three LLMs across disease diagnosis, sentiment analysis, and LLM-as-a-judge classification tasks. We evaluate the learned graphs for predictive fidelity and structural stability, and the MCMC-inspired augmentation for convergence and downstream utility. Our results show that the discovered causal graphs capture meaningful dependencies consistent with LLMs' reasoning. Together, this paper provides a foundation for concept-level explainability of LLMs.

1 Introduction

The paper addresses opaque and often unfaithful LLM explanations by modeling inference as a global causal graph over human-interpretable concepts. Its automated pipeline discovers concepts, expands sparse data with counterfactual chains, and evaluates graph fidelity and stability across tasks.

  • Motivation: LLM inference is opaque because decision factors are unobservable and generated explanations may lack faithfulness.This opacity creates concerns for high-stakes applications such as medicine and law.
  • Motivation: The paper fills a gap in explainability methods that lack a global causal account of how models organize concepts into predictions.Prior counterfactual approaches typically quantify local factor effects without recovering high-level inference.
  • Approach: The proposed graph connects input text to LLM-perceived concept states and then to the prediction, making concept organization and predictive reduction explicit.Unlike graphs modeling real-world variables, this graph represents the LLM’s inference process itself.
  • Approach: The four-phase pipeline generates predictions, extracts class-discriminative concepts, expands the concept space with MCMC-inspired counterfactuals, and constructs a causal graph.The method is model-driven, using the target LLM to identify concepts and generate causal-discovery data.
  • Evaluation: Experiments evaluate three LLMs on sentiment analysis, disease diagnosis, and LLM-as-a-judge classification using predictive fidelity, stability, and expansion utility.The evaluation tests whether discovered parents predict each node and whether expansion improves graph accuracy and stability.
  • Findings: Structured synthetic tasks yield similar explanatory concepts across models, whereas naturalistic data produce distinct latent heuristics; expansion reaches distributional and topological convergence.Discovered causal parents are stronger predictors of nodes than alternative concept sets.

2 Related Work

Related work uses causal graphs for internal components, high-level causal effects, or real-world mechanism discovery, but not for globally reconstructing an LLM’s concept-level reasoning. This paper instead infers the full text-to-concepts-to-prediction graph and expands sparse concept data with counterfactuals.

  • Causal Graphs and Interpretability: Interpretability methods such as attribution, attention, probing, concepts, and rationales are described as fundamentally associative rather than causal.The paper positions causal evidence as necessary for faithful explanations.
  • Causal Graphs and Interpretability: Mechanistic interpretability defines causal graphs over internal components such as attention heads, neurons, residual-stream directions, or higher-level representations.These graphs retain components hypothesized to mediate targeted behavior while abstracting away from complete computation.
  • Causal Graphs and Interpretability: Other causal graphs estimate high-level concept effects on predictions, often assuming a domain-expert graph describing concepts →text →prediction.These approaches generally focus on causal-effect estimation rather than recovering the model’s complete inference structure.
  • Novelty: The paper introduces causal graphs as explanations of model reasoning through the text →concepts →prediction structure, with concepts and relations inferred rather than assumed.This high-level graph utility had received no attention in the cited literature.
  • Causal Discovery and LLMs: LLM-assisted causal discovery typically targets real-world causal edges, and studies report that models may memorize frequent relations instead of showing causal generalization.This goal differs from characterizing the LLM’s own inference process.
  • Causal Discovery and LLMs: Compared with COAT, the paper recovers the full text-to-prediction graph instead of a local Markov blanket and adds counterfactual augmentation for sparse latent decision spaces.The augmentation is intended to support more robust graph discovery.

3 Method

The method constructs a concept-level causal graph of an LLM’s classification inference through a four-phase, model-driven pipeline. It discovers discriminative concepts, expands sparse observations with counterfactual chains, and performs causal discovery over the resulting concept space.

  • The framework recovers a causal graph explaining how a target LLM derives predictions from human-interpretable concepts.
  • The target LLM supplies predictions, latent concepts, concept vectors, and counterfactual texts, so the graph reflects the model’s perspective.Ground-truth labels are replaced with the LLM’s predictions before downstream phases.
  • The four phases are label prediction, discriminative concept extraction, MCMC-inspired counterfactual expansion, and causal discovery.The final graph is constructed over the input text, concept variables, and model prediction.
  • Each concept state records whether the concept is absent and which task classes the model perceives it to support.States correspond to subsets of the task classes, including absence or presence without class discrimination.
  • Counterfactual expansion densifies the concept space by iteratively proposing text changes that target one concept while keeping other concepts stable.Proposals are accepted only when they achieve the target alignment and keep non-target concept drift within a tolerance; failed proposals receive recursive refinement.

4 Experimental Setup

The experiments evaluate three LLMs on disease diagnosis, sentiment analysis, and LLM-as-a-judge classification tasks. The judge setting uses separate query-level datasets and graphs to capture preference reasoning within each fixed topical context.

  • The framework is evaluated with Gemini-2-Flash, Qwen3-14B, and gpt-OSS-20b.Classification uses temperature τ = 0, while concept extraction and counterfactual generation use τ = 0.5.
  • The evaluation covers Disease Diagnosis, Sentiment Analysis, and LLM-as-a-Judge classification.The datasets are LIBERTY, IMDB, and a preference-classification task, respectively.
  • The LLM-as-a-Judge setting constructs a separate causal graph for each query from diverse response pairs.This query-level design captures how the model adjudicates responses within a fixed topical context.

5 Results

The evaluation finds that causal-graph parent sets predict LLM concepts and outputs well, while counterfactual expansion improves predictive accuracy and graph stability. Across tasks, learned concepts and reasoning topologies vary by model on naturalistic data but align more closely on synthetic disease diagnosis.

  • Predictive Fidelity: Graph-based parent predictors outperform average alternative concept subsets across all models, datasets, and target types.The comparison uses average 10-fold cross-validation, with logistic regression predicting either the task output or concept values.
  • Predictive Fidelity: The causal-graph parent set ranks among the top three predictors in most cross-validation cases in every setting but one.It is compared against tens of alternative concept combinations.
  • Causal Graphs: On synthetic Disease Diagnosis, models recover similar concepts and ground-truth variables, whereas naturalistic tasks yield model-specific concepts and reasoning mechanisms.Even similar concepts can produce different causal topologies across models.
  • MCMC Convergence and Stability: The empirical KL-divergence trajectory moves from the orthogonal-expansion upper bound toward the perfect-overlap lower bound during counterfactual expansion.After KL stabilizes, successive causal edge sets have zero Hamming distance, indicating structural stability.
  • Counterfactual Augmentation: Seed + Counterfactuals achieves the highest predictive accuracy across concept nodes and final predictions, while graph parent subsets outperform averages over all concept subsets.Counterfactuals add concept-state combinations absent from the original data, improving dependency estimation.

6 Conclusion

The paper presents concept-level causal graphs as a process-oriented explanation of LLM inference and uses counterfactual expansion to support accurate, stable graph discovery. It also proposes predictive and structural validation protocols for settings without gold-standard reasoning graphs.

  • Conclusion: The framework models LLM inference with causal graphs over human-interpretable concepts.It is intended as an accessible and actionable interpretability representation for model developers and domain experts.
  • Conclusion: MCMC-inspired counterfactual expansion addresses sparse observational coverage in latent concept spaces.Iterative counterfactual generation produces denser and more representative data for causal discovery.
  • Conclusion: Predictive and structural validation protocols demonstrate the quality of discovered graphs and the contribution of MCMC-inspired expansion.These protocols address evaluation without gold-standard reasoning graphs.
  • Conclusion: The proposed direction focuses explainability on modeling inference mechanisms rather than explaining isolated predictions.Future extensions target additional languages, open-ended generation, multimodal domains, and broader evaluation.

7 Limitations

The method’s current limitations concern concept-extraction reproducibility, graph-level validation scope, and reliance on the target LLM for annotation and counterfactual generation.

  • Concept Extraction: Concept candidates may depend on arbitrary example grouping because extraction uses small batches and only one inference pass.Multiple shuffled extraction passes are proposed when computational budget permits.
  • Evaluation Scope: Validation primarily tests local parent-set predictive fidelity and does not directly verify longer multi-hop chains or the recovered graph hierarchy.The protocol establishes that immediate predictors are informative and structurally stable.
  • Self-Annotation: Counterfactual generation relies on the target LLM to annotate concepts, generate edits, and judge constraint satisfaction.Errors in these intermediate steps may propagate despite acceptance checks for target alignment and concept drift.

A.1 Datasets

The evaluation covers three classification tasks using extracted textual content, with dataset scale constrained by computational and API costs. The natural-dataset analysis also addresses positional bias in LLM-as-a-judge classification.

  • Datasets: The evaluation spans Disease Diagnosis, Sentiment Analysis, and LLM-as-a-Judge classification tasks.Sentiment Analysis uses a subsampled IMDB corpus of N = 2096 instances.
  • Datasets: Sentiment Analysis uses raw IMDB movie-review texts with N = 2096 instances and Positive or Negative labels.The corpus is subsampled because of computational and API cost constraints.
  • Datasets: Disease Diagnosis uses raw patient descriptions from a synthetic medical corpus with N = 1448 instances.The task predicts Migraine, Sinusitis, or Influenza.
  • LLM-as-a-Judge: Swapping candidate-response order changed LLM-as-a-judge outcomes in over 30% of instances.The study presents each pair in both orders and runs parallel chains to balance the expanded counterfactual data.
  • Dataset Analysis: Extracted concepts are similar across models on synthetic Disease Diagnosis but more model-specific on natural datasets.The synthetic concepts faithfully reconstruct the base graph used to generate the data.

A.3 Data Expansion

The expansion phase addresses sparse concept coverage by generating counterfactual texts that vary candidate concepts across classification targets. These examples densify the concept distribution used for causal discovery.

  • Cold-start challenge: The LAJ setting starts each graph from a single seed example, making the observed concept distribution insufficient for reliable causal discovery.The evaluation includes Q = 48 Gemini graphs, Q = 18 QWEN graphs, and Q = 10 GPT-OSS graphs.
  • Counterfactual proposals: For a concept aligned with one class, the LLM rewrites the text toward every remaining class; neutral concepts are expanded across all labels.The perturbation direction depends on the concept’s current alignment context.
  • Concept space: Table 2 reports the latent features each model identifies as most differentiative across datasets and tasks.These extracted concepts provide the candidate concept space used by the framework.
  • Counterfactual proposals: The exhaustive perturbation strategy generates counterfactual examples that create a denser joint distribution over concept states.This denser distribution becomes the starting point for the MCMC expansion phase.

A.4 Validation Details

Validation compares causal-parent predictors and graph configurations against alternative feature subsets and seed-based baselines. The results generally favor MCMC-expanded graphs and learned causal parents, with a localized exception.

  • Predictive validation: The validation protocol constructs graphs on 80% training splits and evaluates each target node’s parents against alternative predictor subsets.The procedure uses 10 cross-validation rounds with a 20% held-out validation slice.
  • Predictive validation: Causal-parent predictors achieve the highest classification accuracy across the vast majority of datasets and model architectures.Alternative feature combinations tie in a few isolated cases but never outperform the causal parents.
  • MCMC comparison: The MCMC+Filtered Seed graph combines counterfactual expansions with seed instances that initiated valid chains, while Filtered Seed omits those expansions.These configurations isolate the downstream contribution of MCMC-generated counterfactuals.
  • MCMC comparison: The MCMC+Filtered Seed graph generally outperforms Seed and Filtered Seed configurations for parent-conditioned accuracy across most concepts.A few exceptions occur, including suboptimal intermediate-concept performance for Gemini on LIBERTY.
  • MCMC comparison: The learned graph-based parent configuration consistently achieves top-tier accuracy and shifts the predictive-accuracy distribution upward under MCMC+Filtered Seed.This pattern supports expansion of the observational distribution while preserving structural causal signal.

A.5 MCMC Convergence and Stability

The paper evaluates MCMC-inspired counterfactual expansion using KL-based distributional diagnostics and graph-structure comparisons. Across models and datasets, the expanded data converge distributionally and produce increasingly stable causal graph topologies, with faster convergence for query-specific LAJ graphs.

  • Convergence diagnostics: Classic MCMC diagnostics are unavailable because the latent-concept distribution is unknown and only one chain runs per text example.The method therefore uses a tailored KL-based convergence metric instead of variance-based diagnostics.
  • Convergence diagnostics: The method tracks a global empirical probability vector over all m^|C| concept combinations and compares successive distributions using KL divergence.The vector indexes concept-label combinations and records their empirical probabilities.
  • Theoretical bounds: Perfect-overlap and orthogonal-expansion scenarios define lower and upper KL boundaries for distinguishing stabilization from continued exploration.The orthogonal bound uses smoothing ϵ = 10^-10 for previously empty bins.
  • Empirical convergence: Across examined models and datasets, empirical KL curves move away from the exploration bound and toward the optimal boundary before chain termination.This pattern is reported as robust and stable convergence of the MCMC data expansion.
  • Dataset-specific behavior: LAJ converges faster because each query uses a smaller, bounded combinatorial state space, while IMDB and LIBERTY use a unified global concept space.LAJ displays separate convergence trajectories for query-specific causal graphs rather than one dataset-level curve.
  • Structural stability: For Gemini and Qwen on IMDB and LIBERTY, structural Hamming distances between successive consensus graphs decrease and eventually reach zero.Consensus graphs are learned from 10 cross-validation runs at each expansion milestone, with edges retained by majority vote.

B Running Example: Full Pipeline Walkthrough

The toy papaya task illustrates the complete pipeline from LLM label prediction and discriminative concept discovery through counterfactual expansion and σ-CG graph construction. The surrounding evaluations compare predictive accuracy, concept predictability, MCMC convergence, and graph stability across models and datasets.

  • Phase 1–2: Prediction and concept discovery: The pipeline first replaces dataset labels with the target LLM’s predictions, then extracts concepts that distinguish those predicted classes.In the example, the LLM labels a mushy papaya not-tasty and proposes Softness, Color, and Origin.
  • Phase 2: Concept filtering: Concepts are retained when they are relevant and discriminative in at least τ = 0.5 of annotated examples in the binary toy task.Origin fails relevance when usually absent, while concepts aligned with both classes fail discriminativeness.
  • Phase 2: Concept representation: Each example becomes a concept vector linking textual features to class-aligned states, such as mushy→Softness={not-tasty} and bright orange→Color={tasty}.The representation records how the LLM associates each discovered concept with the task classes.
  • Phase 3: Counterfactual expansion: MCMC expansion selects a concept and target class, chooses MORE or LESS, and prompts the LLM to generate a counterfactual while preserving other concepts.The example targets Softness, samples tasty, and requests a MORE shift.
  • Phase 3: Acceptance: A proposal is accepted when the target concept aligns with the sampled class and non-target concept changes remain within the allowed error threshold.The papaya proposal passes target alignment and has nerr = 0, so it is appended to Dmcmc.
  • Phase 4: Causal discovery: σ-CG receives the expanded concept vectors and LLM labels, using the text as a root and the prediction as a sink to output a directed graph.The toy graph visualizes dependencies among Softness, Color, and the LLM prediction.
  • Evaluation: Figure 6 compares bootstrap accuracy distributions for graph-specified parent sets against alternative predictor subsets across 10 cross-validation rounds.Light blue denotes causal-parent conditioning, while purple denotes alternatives that omit at least one required parent.

G Computational Resources and Compute Cost

The computational bottleneck is repeated LLM sampling during MCMC-inspired counterfactual expansion. Costs vary substantially by model access, while downstream causal discovery is lightweight and runs on standard CPU infrastructure.

  • Bottleneck: Repeated, iterative LLM sampling for counterfactual expansion is the framework’s primary computational bottleneck.Concept extraction and causal discovery are less computationally demanding than the expansion stage.
  • API cost: $540 USD was the approximate Gemini API cost for the complete concept-extraction and MCMC-expansion pipeline.This estimate covers the full pipeline executed through the Gemini API.
  • Local compute: Under $5 USD in equivalent compute time was incurred for Qwen3 and gpt-OSS counterfactual generation on the institutional GPU cluster.The reported marginal compute cost applies to the open-weight models.
  • Downstream processing: After counterfactual datasets are generated, causal discovery is computationally lightweight and can run on standard CPUs within minutes.The downstream phase trains multinomial logistic regression models to extract graph topologies.
Loading 2606.05972v1…