PaperlayerView exact revision on arXiv

Source-linked AI summary

Score-Based Causal Discovery of Latent Variable Causal Models

Ignavier Ng, Xinshuai Dong, Haoyue Dai, Biwei Huang, Peter Spirtes, Kun Zhang

arXiv:2605.20396v1cs.LGstat.ML

TL;DR

Latent variables make causal discovery difficult because constraint-based methods can face empirical limitations and score-based formulations are challenging. The paper develops SALAD with identifiable scoring functions, degrees-of-freedom characterizations, and exact or continuous search methods. The proposed methods achieve score equivalence and consistency, improve performance over existing constraint-based methods, and provide a unified view of related approaches.

  • Problem

    Causal discovery with causally related latent variables lacks suitably formulated score-based methods with identifiability guarantees, while existing constraint-based methods face empirical challenges.

  • Method

    SALAD combines degrees-of-freedom characterizations with exact and continuous score-based methods for latent variable causal models.

  • Results

    The scoring formulation is score equivalent and consistent, exact methods asymptotically identify the true equivalence class, and experiments report improved performance over existing constraint-based methods.

  • Takeaways & Limitations

    SALAD provides a unified score-based framework for latent variable causal discovery across several structural assumptions and existing constraint-based methods.

  • Takeaways & Limitations

    The exact methods require relatively long runtimes, and theoretical justification for using BIC under the considered latent-variable assumptions remains future work.

Abstract

from arXiv · show

Identifying latent variables and the causal structure involving them is essential across various scientific fields. While many existing works fall under the category of constraint-based methods (with e.g. conditional independence or rank deficiency tests), they may face empirical challenges such as testing-order dependency, error propagation, and choosing an appropriate significance level. These issues can potentially be mitigated by properly designed score-based methods, such as Greedy Equivalence Search (GES) (Chickering, 2002) in the specific setting without latent variables. Yet, formulating score-based methods with latent variables is highly challenging. In this work, we develop score-based methods that are capable of identifying causal structures containing causally-related latent variables with identifiability guarantees. Specifically, we show that a properly formulated scoring function can achieve score equivalence and consistency for structure learning of latent variable causal models. We further provide a characterization of the degrees of freedom for the marginal over the observed variables under multiple structural assumptions considered in the literature, and accordingly develop both exact and continuous score-based methods. This offers a unified view of several existing constraint-based methods with different structural assumptions. Experimental results validate the effectiveness of the proposed methods.

1. Introduction

The paper develops SALAD, score-based methods for latent variable causal models, addressing limitations of constraint-based discovery and providing identifiability guarantees. It characterizes observed-variable degrees of freedom and introduces exact and continuous search procedures.

  • Motivation: Latent confounders can create spurious observed correlations, making causal discovery methods that assume causal sufficiency unreliable.The paper motivates latent-variable models with examples involving psychological measurements and unstructured data.
  • Limitations of prior methods: Constraint-based methods for latent variables can suffer from overgeneralized PAGs, testing-order dependency, and error propagation.These methods iteratively match statistical properties to candidate structures.
  • Contributions: SALAD develops score-based methods that identify causal relations among latent variables and provide a unified view of several existing constraint-based approaches.The framework targets latent variable causal models under structural assumptions studied in prior work.
  • Contributions: A properly formulated scoring function is score equivalent and yields a structure algebraically equivalent to the true structure, preserving observed equality constraints.These constraints include conditional independence and rank deficiency constraints.
  • Contributions: Exact methods asymptotically identify the true equivalence class, while continuous methods improve computational efficiency in some settings.The methods estimate the whole causal structure, including relations among latent variables.
  • Empirical validation: Experiments report improved performance over existing constraint-based methods for estimating latent variable causal structures.The experiments are presented as validation of score-based methods.

2. Latent Variable Causal Models

The paper models measured and latent variables with a linear latent variable causal model whose structure is encoded by coefficient supports. It formulates observed covariance, handles latent-noise indeterminacy, and estimates structure up to latent-variable relabeling.

  • Model: The model contains measured variables X and latent variables L connected through a DAG and generated by linear structural equations.The latent variables follow L = CL + EL, and measured variables follow X = BL + EX.
  • Model: The support of B encodes edges from latent variables to measured variables, while the support of C encodes edges among latent variables.C and its binary adjacency representation are assumed strictly upper triangular without loss of generality.
  • Covariance formulation: The observed covariance is determined by the latent and measured-variable structural parameters together with noise covariance matrices.The formulation explicitly distinguishes population covariances of measured and latent variables from noise covariances.
  • Identifiability: Latent-variable labels are generally unidentifiable, so structures are compared up to Markov equivalence after relabeling latent variables.The estimation goal is to recover the structure up to a specified model equivalence.
  • Identifiability: Latent-noise covariance is indeterminate from the observed covariance, allowing estimation to assume ΩL = I without loss of generality.Alternative parameters with the same structural supports can generate the same observed covariance.
  • Likelihood: The likelihood is formulated using the empirical covariance matrix while fixing the latent-noise covariance according to the indeterminacy result.This provides the basis for score-based structure estimation.

3. Score-Based Identification of Latent Variable Causal Models

The paper formulates score-based identification for linear latent variable causal models by selecting structures that generate observed covariance matrices with minimal degrees of freedom. Under generalized faithfulness, this yields algebraic equivalence generally, while score equivalence and structural assumptions enable finer identification guarantees.

  • Distribution sets and constraints: A structure’s distribution set contains the observed covariance matrices generated by varying its model parameters, and is a semialgebraic set.The model imposes equality constraints including conditional independence, rank deficiency, and possibly Verma constraints.
  • Scoring function: The scoring procedure searches for the structure with the smallest degrees of freedom that can generate the empirical covariance matrix.Structures unable to generate the covariance matrix receive infinite score, and score equivalence permits searching over Markov equivalence classes.
  • Algebraic equivalence: Under generalized faithfulness, minimizing scoredim yields a structure algebraically equivalent to the true DAG in the large sample limit.The estimated and true structures therefore impose the same equality constraints on the marginal distribution of measured variables.
  • From algebraic to structural identification: Structural assumptions can translate algebraic equivalence into finer notions of model equivalence, including Markov equivalence.The paper emphasizes that relating algebraically equivalent structures to the true structure is difficult without such restrictions.
  • BIC: BIC performs strongly empirically in the studied settings, although its standard theoretical justifications may not apply directly to latent-variable models.The paper identifies theoretical justification of BIC under these structural assumptions as a direction for future work.
  • Score equivalence: The proposed scoring functions are score equivalent, so Markov-equivalent structures receive identical scoredim and BIC scores.This avoids recomputing scores for every DAG within the same Markov equivalence class.

4. Linear 1-Factor Latent Variable Models

Under the one-factor latent-variable assumption, the paper characterizes model degrees of freedom and proves that exact score-based search can recover the true Markov equivalence class under generalized faithfulness. It also develops a continuous formulation to reduce the computational burden of evaluating many candidate structures.

  • Assumption: Each measured variable has one latent parent, while each latent variable has at least three measured children under the Silva et al. assumption.The paper notes that two measured children may suffice in its setting.
  • Degrees of freedom: Under this assumption, the model dimension equals the number of DAG edges plus the number of measured variables: dim(G) = |G| + m.The edges include latent-to-latent and latent-to-measured edges.
  • Consistency and exact search: Under generalized faithfulness, the optimal-scoring structure is Markov equivalent to the true structure in the large sample limit.This establishes the correctness of the exact score-based approach under the one-factor structural assumption.
  • Exact search: Exact search enumerates non-Markov-equivalent latent DAGs and ordered partitions of measured variables, retaining structures that satisfy the assumption.The enumeration avoids redundant structures because score-equivalent candidates have the same score.
  • Continuous search: Exact search can be computationally intensive because it computes a score for every structure, motivating a unified continuous optimization procedure.The continuous formulation uses support matrices and constraints encoding the one-parent and minimum-three-children requirements.
  • Continuous optimization: Gumbel-Sigmoid and Gumbel-Softmax are used to estimate binary support matrices within the continuous optimization procedure.Slack variables convert inequality constraints into equality constraints before optimization.

5. Linear Latent Hierarchical Structures

Under Huang et al.'s more general structural assumption, the paper develops degrees-of-freedom calculations and an exact score-based search for latent hierarchical structures. Under generalized faithfulness, the optimal score recovers the true structure up to specified equivalence operators in the large-sample limit.

  • Structural assumptions: The generalized assumption allows measured variables to have multiple latent parents and accommodates latent variables without measured children.It is more general than the earlier measurement-tree assumption.
  • Structural assumptions: Latent atomic covers and pure-child sets provide the structural building blocks used to characterize identifiable latent hierarchical graphs.Pure-child sets require observed children with exactly the latent set as parents and no latent descendants.
  • Degrees of freedom: Algorithm 2 computes an upper bound on degrees of freedom by iterating over subsets of latent variables with shared parents and children.The authors conjecture, based on 10,000 simulations and experiments, that this bound is tight under Omin(Oskeleton(·)), but do not prove it.
  • Degrees of freedom: k latent variables sharing parents and children reduce the dimension upper bound to |G|+m−k(k−1)/2 when either the parent or child count is at least k.The reduction reflects redundant edges that can be removed without changing the generated distribution set.
  • Exact score-based search: The exact method enumerates non-Markov-equivalent structures, computes their degrees of freedom, and selects the structure with the optimal score.The search is bounded by a maximum latent-variable count and retains only structures satisfying Assumption 3.
  • Exact score-based search: Under Assumption 3 and generalized faithfulness, the optimal score is Markov equivalent to the true hierarchical structure up to rank-equivalent graph operators.The theorem applies in the large-sample limit to the atomic-cover representation of the estimated and true structures.

6. Experiments

Experiments compare SALAD and SALAD-CS with FOFC, HUANG, and GIN on 1-factor and hierarchical structures across sample sizes. The proposed methods achieve higher skeleton F1 scores and MEC SHDs, especially with small samples.

  • Experimental setup: The experiments compare exact SALAD and continuous SALAD-CS against FOFC, HUANG, and GIN using BIC scoring.FCI is excluded because it would produce largely uninformative complete PAGs for the considered ground truths.
  • Experimental setup: The evaluation uses 1-factor and hierarchical ground truths, sample sizes from 100 to 10,000, skeleton F1, and MEC structural Hamming distance.Each ground truth receives three random trials, with means and standard deviations reported.
  • Results: The proposed methods achieve higher F1 scores and SHDs than the baselines, particularly at small sample sizes.At large sample sizes, their F1 scores approach one under both structural assumptions.
  • Results: 0.99 and 0.92 average F1 scores are achieved by SALAD at 100 samples for 1-factor and hierarchical structures, versus 0.90 and 0.57 for the second-best baseline.FOFC's 0.90 result for the 1-factor model included four invalid runs.

7. Conclusion and Discussion

The work proposes SALAD, a score-based method for identifying causal relations among latent variables, and develops score-based tools with identifiability guarantees. Its exact methods have relatively long runtimes, motivating more efficient future search procedures.

  • SALAD identifies causal relations among latent variables using a score-based causal discovery method.
  • The framework achieves score equivalence and consistency, characterizes degrees of freedom, and provides exact and continuous score-based methods.
  • The methods unify multiple existing constraint-based approaches for latent-variable causal models and validate score-based discovery experimentally.
  • Exact methods require relatively long runtimes, while future work includes greedy approaches for more efficient and scalable search.

Supplementary Material

The supplementary material develops theoretical foundations for score-based learning with latent variables, including covariance-based equivalence, score equivalence, algebraic-equivalence consistency, and degrees-of-freedom results. It also discusses modeling assumptions and their relation to latent-variable structures.

  • Latent-variable models represent measured variables as effects or indirect proxies of latent variables, including in psychometrics and fMRI analysis.
  • The supplementary discussion compares structural assumptions, including requirements on observed variables, latent-variable children, and hierarchical latent structures.
  • Under Markov equivalence, the proposed dimension and BIC scores assign equal scores to equivalent DAGs.
  • Under generalized faithfulness, minimizing the dimension-based score recovers a DAG algebraically equivalent to the true DAG in the large-sample limit.
  • For structures satisfying Assumption 2, the degrees of freedom equal |G| + m, based on generic identifiability of the parameter-to-covariance map.

B.5. Proof of Theorem 2

The proof of Theorem 2 shows that, under generalized faithfulness and Assumption 2, global minimization of the proposed score recovers the true latent-variable DAG up to Markov equivalence and latent-variable relabeling.

  • Theorem 2 assumes generalized faithfulness and a true DAG satisfying Assumption 2, then minimizes the proposed score over structures satisfying that assumption.
  • The estimated DAG and the true DAG generate the same covariance distribution and have equal algebraic structure in the large-sample limit.
  • The measurement models of the estimated and true DAGs are identical up to relabeling of latent variables.
  • Their latent-variable structural models are Markov equivalent, so the complete estimated and true DAGs are Markov equivalent up to latent-variable relabeling.

B.6. Proof of Proposition 3

The proof establishes how orthogonal transformations can remove redundant edges without changing the represented distribution, yielding degrees-of-freedom bounds for repeated latent-variable neighborhoods. The argument extends this reduction through hierarchical groups under Assumption 3.

  • The distribution set M(G; ΩL = I) is defined by the linear latent-variable covariance parameterization with support constraints on B and C.
  • An orthogonal transformation of B and C can preserve the generated covariance matrix, enabling equivalent parameterizations with modified edge supports.
  • If k latent variables share parents and children and have at least k parents or children, k(k − 1)/2 edges can be removed without changing the distribution set.
  • Under the same repeated-neighborhood condition, the degrees of freedom satisfy dim(G) ≤ |G| + m − k(k − 1)/2.
  • For Assumption 3, Algorithm 2 outputs an upper bound on dim(G) by iteratively removing distribution-preserving edges across ordered latent-variable groups.

B.8. Definition of Graph Operators and Proof of Theorem 3

The section defines three graph operators for latent causal structures and proves that the score-based optimization recovers the true structure up to the resulting equivalence. These operators preserve rank constraints, so rank-based methods can identify structures only up to them.

  • Graph operators: The minimal-graph operator merges latent atomic covers under specific pure-child, cardinality, and atomic-cover conditions.
  • Graph operators: The skeleton operator adds edges from an atomic cover to children not already covered by contained atomic covers.
  • Graph operators: The intra-atomic operator adds edges within an atomic cover so that its latent variables form a fully connected DAG.
  • Graph operators: A worked example applies the skeleton, minimal-graph, and intra-atomic operators sequentially to transform the original graph.
  • Correctness and identifiability: The three operators preserve rank constraints, limiting rank-based causal discovery to identification up to these operators.
  • Correctness and identifiability: Theorem 3 establishes that the global score minimizer is Markov equivalent, after atomic transformation, to the transformed true structure under generalized faithfulness and the stated structural constraints.

C. Supplementary Experiment Details

The supplementary experiments specify optimization, baseline, evaluation, and validity procedures for comparing latent causal discovery methods. They evaluate structural recovery using permutation-invariant metrics and track invalid FOFC runs.

  • Implementation: The exact-search implementation uses L-BFGS from SciPy and PyTorch with default hyperparameters to solve the BIC optimization problem.
  • Baselines: GIN, HUANG, and RCD use publicly available implementations with default hyperparameters, while FOFC uses py-causal with a Wishart test at significance level 0.001.
  • Metrics: SHDs are computed over Markov equivalence classes and latent-variable permutations, while skeleton F1 scores select the highest value across permutations.
  • Validity: FOFC experiments additionally report the number of valid runs because errors occurred in some experimental runs.

D. Runtime and Computational Efficiency

The runtime analysis shows that SALAD and SALAD-CS are substantially slower than several baselines, although their performance is reported as better than RCD and improved over faster methods. The exact-search implementation is presented as a small-scale proof of concept, with scalability left for future work.

  • Runtime: 8.77 ± 0.73 and 44.88 ± 8.02 minutes are SALAD runtimes for 10 and 11 measured variables in 1-factor models, respectively.
  • Runtime: 16.11 ± 2.01 minutes is SALAD’s runtime for hierarchical structures, while SALAD-CS takes 14.17 ± 0.69 and 14.73 ± 1.91 minutes for 10 and 11 variables.
  • Performance trade-off: SALAD and SALAD-CS have comparable runtime to RCD but better performance, while GIN, HUANG, and FOFC generally finish within one minute.
  • Scalability: Future efficiency improvements include GPU acceleration and concurrent score computations across CPUs, with greedy GES-like search proposed for greater scalability.
  • Scalability: The methods’ relatively long runtime reflects the known computational cost of exact score-based search, and the current procedure is intended for relatively small numbers of variables.
Loading 2605.20396v1…