PaperlayerView exact revision on arXiv

Source-linked AI summary

LUCID: Learning Under Confounding for Inference and Discovery in Time Series

Mohammad Fesanghary

arXiv:2609.31315v1cs.LGstat.ML

TL;DR

Unobserved common causes can create spurious associations that causal discovery mistakes for direct edges, while sparse and pervasive confounding may require different corrections. LUCID routes data to a regime-matched deconfounding layer that can wrap existing discovery engines. Across ten out-of-distribution confounding families, it achieves 0.596 family-weighted directed, lag-resolved F1 versus 0.404 for the strongest unwrapped baseline.

  • Problem

    Latent common causes induce spurious associations that can be mistaken for direct effects, and corrections suited to pervasive confounding may perform poorly for sparse confounding.

    Source: §1 · p. 1

  • Method

    LUCID uses a data-derived spectral router to select a regime-matched correction, including spectral attenuation and lag-0 recovery for pervasive confounding, while wrapping rather than replacing discovery engines.

    Source: §1 · pp. 1–2, §6 · p. 8

  • Results

    0.596 versus 0.404 for the strongest unwrapped baseline in family-weighted directed, lag-resolved F1 across ten out-of-distribution confounding families.

    Source: §6 · p. 8

  • Takeaways & Limitations

    LUCID’s gains persist under transient and heavy-tailed confounding, and the layer also improves PCMCI+ and NTS-NOTEARS when wrapped around them.

    Source: §6 · p. 8

    Spectral trimming can attenuate genuine lag-0 signal that overlaps factor directions, and weak or nonlinear factor structure can lead the router to misclassify the regime.

    Source: §6 · p. 8

Abstract

from arXiv · show

Unobserved common causes are pervasive in real-world time series and can induce spurious associations that causal discovery methods mistake for direct edges. We propose LUCID (Learning Under Confounding for Inference and Discovery, a regime-adaptive deconfounding layer that first estimates the confounding regime from data using a Mar\v{c}enko--Pastur spectral router, then applies a deconfounding strategy matched to that regime. When the spectrum indicates pervasive factor confounding, LUCID attenuates factor-dominated variation and recovers contemporaneous (lag-$0$) structure from the resulting innovations, with edge selection calibrated against a data-driven edge-free null. Rather than being tied to a particular discovery algorithm, it can wrap existing discovery engines; we demonstrate consistent improvements across three such methods. On a diverse synthetic out-of-distribution benchmark spanning changes in confounder strength and sparsity, loading density, lag structure, volatility dynamics, edge heterogeneity, persistence, intermittency, and tail behavior, LUCID achieves the best family-weighted directed, lag-resolved graph $F_1$ ($0.60$), improving over the strongest baseline by $0.19$ absolute ($\approx\!46\%$ relative). Its advantage widens relative to looser lag-collapsed scoring, and remains robust under intermittent and heavy-tailed confounding. Code reproducing the method, the benchmark generators, and every reported experiment is available at https://github.com/bloomberg/causal-ts.

1. Introduction

LUCID addresses latent confounding by routing each time series to a regime-matched correction, rather than replacing the causal discovery engine. Its benchmark results show gains in directed, lag-resolved graph recovery and transfer across discovery methods.

1. Introduction

  • Sparse and pervasive confounding can leave different or weak statistical signals, so corrections suited to one regime may degrade the other.Pervasive factors can create low-rank covariance structure; a sparse latent fork may produce neither a separated eigenvalue nor informative rank constraints.Source: §1 · p. 1
  • LUCID uses a data-derived Marčenko–Pastur spectral statistic to route each dataset to a correction for sparse or pervasive confounding.The threshold depends on data dimension and length rather than being tuned per dataset.Source: §1 · p. 1
  • The layer acts on data and returned graphs, not discovery-search internals, and is demonstrated with CDNOTS, PCMCI+, and NTS-NOTEARS.The reported gains extend to the latter two engines when LUCID wraps them.Source: §1 · pp. 1–2
  • Routing recovers most of the per-dataset oracle’s value (regret 0.007) and avoids the 0.26 family-weighted F1 loss of always choosing the sparse branch.The router is intended to provide robustness without knowing the confounding regime in advance.Source: §1 · p. 2
  • On the pervasive branch, LUCID attenuates factor-dominated variation and uses a data-calibrated edge-free null to recover contemporaneous candidates.A persistence gate suppresses co-parent moralization edges that precision-matrix support could admit.Source: §1 · p. 2
  • 0.601 versus 0.411 for the strongest baseline on strict directed, lag-resolved F1; LUCID’s margin grows relative to lag-collapsed scoring.Across ten confounding families, wrapping LUCID around PCMCI+ or NTS-NOTEARS also improves each method.Source: §1 · p. 2

2. Related Work

Related work addresses temporal causal discovery, latent confounding, and factor structure through distinct methods and recovered graph representations. LUCID instead adapts deconfounding to the observed regime while leaving the downstream discovery engine interchangeable.

2. Related Work

  • Temporal discovery methods span conditional-independence pruning, global score or continuous optimization, and nonlinear predictive approaches.Examples include PCMCI+, CDNOTS, NTS-NOTEARS, VARLiNGAM, Neural Granger Causality, NAVAR, and CUTS+ (Runge et al., 2019; Runge, 2020; Sadeghi et al., 2025; Sun et al., 2023; Hyvärinen et al., 2010; Tank et al., 2021; Bussmann et al., 2021; Cheng et al., 2024).Source: §2 · p. 2
  • Latent-confounder methods include PAG-based approaches, rank constraints, and recovery of latent components for downstream discovery.These approaches represent or reconstruct hidden structure, whereas LUCID targets a directed, lag-resolved graph over observed variables while treating confounding as a nuisance.Source: §2 · p. 2
  • Factor-based methods exploit low-rank or algebraic signatures of pervasive confounding, but some spectral relatives target i.i.d. data rather than time series.LUCID’s S−L step recovers contemporaneous structure rather than discarding lag 0, using a cheaper spectral procedure than the convex low-rank-plus-sparse program it draws on.Source: §2 · pp. 2–3
  • LUCID tests whether the spectrum supports pervasive confounding before applying factor-style correction, then separates that correction from causal search.Its S−L branch attenuates dominant factor directions and reconstructs lag-0 candidates using a data-calibrated edge-free null and marginal corroboration.Source: §2 · p. 3

3. Background and Notation

The paper represents temporal causal structure as a lag-indexed graph and evaluates directed edges with a metric that requires the cause, effect, and lag all to match. Its background distinguishes pervasive factor confounding from localized sparse confounding.

3. Background and Notation

  • The lagged causal graph G is indexed by cause, effect, and lag, with G[c,e,ℓ] = 1 denoting a direct effect from cause c to effect e at lag ℓ.The series is represented by observed variables over time, with lags from 0 through L.Source: §3 · p. 3
  • Directed, lag-resolved F1 counts an edge as correct only when its cause, effect, and lag match the ground truth.Same-variable autoregressive edges are excluded from scoring; pair-level F1 instead collapses across lags and can credit a correct variable pair at the wrong delay.Source: §3 · p. 3
  • Pervasive confounding occurs when a few latent factors affect many observed variables, while a sparse latent cause affects only a few and need not leave a spectral signature.LUCID uses this distinction to route between regimes.Source: §3 · p. 3

4. Method

LUCID routes data to a regime-matched deconfounding correction while keeping the same base discovery engine. On the pervasive branch, it attenuates factor-dominated variation and reconstructs contemporaneous candidates using spectral recovery.

4. Method

  • LUCID infers the confounding regime, runs the same base discovery engine on either branch, and applies a branch-specific correction.The pervasive branch replaces the contemporaneous slice with S−L recovery.Source: §4 · p. 3

4.1. Regime routing via the Marˇcenko–Pastur null

  • The router compares the top-two residual spectral variance fraction with a Marčenko–Pastur-calibrated threshold, selecting sparse when R ≤ τ and pervasive otherwise.The cutoff adapts to d and T; k = 2 and γ = 1.3 are fixed across datasets.Source: §4.1 · p. 3

4.2. Regime-specific discovery and filters

  • The sparse branch applies PDS to every candidate edge, while the pervasive branch retains lagged edges and reconstructs the lag-0 slice with S−L.LUCID’s contribution is the regime-adaptive layer, MP-based routing, and S−L edge-free-null calibration and persistence gate; the base engine and PDS filter are adopted components.Source: §4.2 · pp. 3–4

4.3. Contemporaneous recovery via spectral deconfounding

  • Pervasive same-period factors can create dense lag-0 dependence, so S−L attenuates dominant factor directions and recovers sparse conditional-dependence candidates instead of dropping the slice.The spectral procedure is inspired by low-rank-plus-sparse graphical-model methods but does not solve their convex program or inherit its guarantees.Source: §4.3 · p. 4
  • A persistence gate requires marginal corroboration alongside conditional association, suppressing weakly supported pairs, but co-parents may still survive as lag-0 skeleton candidates.Lag-0 orientations use lead-lag asymmetry in the observed series, a heuristic rather than an identification guarantee.Source: §4.3 · pp. 4–5
  • S−L thresholds edge scores against a data-derived edge-free max-statistic null; under the complete null and shift-exchangeability, the randomization procedure targets family-wise error α.The stated guarantee does not rule out moralization errors when genuine lag-0 edges exist; a stationary block-bootstrap null is offered for strong conditional heteroskedasticity.Source: §4.3 · p. 5

4.4. Fixed hyperparameters

  • LUCID hyperparameters are fixed across datasets, benchmarks, and the CDS case study, with no dataset-specific tuning.The persistence width is selected on lag-0 generators excluded from evaluation; reported results are stable across tested parameter ranges.Source: §4.4 · p. 5

5. Experiments

Across the out-of-distribution benchmark, LUCID improves directed, lag-resolved graph F1 over unwrapped baselines and transfers across discovery engines. Router diagnostics, stress tests, and a market-calibrated semi-synthetic case study examine its scope and behavior.

5.1. Out-of-distribution generalization

  • The equally weighted benchmark spans ten confounding families, including sparse topologies and pervasive settings varying volatility, loading density, lag structure, edge heterogeneity, and persistence.The dense-loading family tests strong factor footprints alongside genuine contemporaneous edges.Source: §5.1 · p. 5
  • Under stricter lag-resolved scoring, LUCID loses 0.018 from 0.619, while baselines lose 0.04–0.08, including 0.082 for LPCMCI and 0.075 for PCMCI+.The authors report that LUCID more often places an edge at the correct lag, not merely the correct variable pair.Source: §5.1 · p. 5

5.2. LPCMCI and TS-ICD: latent-confounder-aware baselines

  • LPCMCI and TS-ICD return PAGs whose projection to binary lag-resolved edges discards information their native output preserves, limiting how completely their F1 scores assess them.LPCMCI was not run on pervasive-dense because orientation exceeded the 10 GB memory budget; it was reported on the other nine families.Source: §5.2 · p. 6

5.3. Router diagnostics

  • At γ = 1.3, the router detects all 25 pervasive diagnostic cases and routes 10 of 30 sparse cases to the pervasive branch; on a separate rerun, its F1 is 0.592 versus 0.600 for a per-dataset oracle.The router’s oracle regret is 0.007, and automatic routing avoids the 0.337 F1 of always choosing the sparse branch.Source: §5.3 · p. 6
  • Unconditionally filtering pervasive-branch lagged edges reduces family-weighted F1 from 0.592 to 0.451, so the correction is restricted to lag 0.LUCID leads at four of five tested factor duty cycles, improves from 0.623 under full persistence to 0.818 at duty cycle 0.05, and leads across eight tested noise distributions.Source: §5.3 · pp. 6–7

5.1. Out-of-distribution generalization

  • 0.601±0.025 family-weighted directed, lag-resolved F1 for LUCID with CDNOTS compares with 0.411 for the strongest unwrapped baselines, a 0.19 absolute gain.Wrapped PCMCI+ reaches 0.596±0.025 and wrapped NTS-NOTEARS 0.538±0.023; wrapped engines improve in 27 of 30 engine×family comparisons.Source: §5.1 · p. 5, Table 1 · p. 7
  • Pooled results show a precision-driven gain: LUCID’s precision is 0.491–0.614 at recall 0.571–0.615, while non-PAG baselines have precision 0.130–0.287 at recall 0.596–0.721.LUCID’s SHD is 13.4–14.5 versus 27.1–67.1 for those baselines; PAG methods differ in recall and graph sparsity.Source: §5.1 · p. 6, Table 2 · p. 7

5.4. A market-calibrated semi-synthetic case study

  • In the market-calibrated CDS semi-synthetic case study, wrapping PCMCI+ yields 0.441 versus 0.093 for the strongest unwrapped baseline at T = 1000, with unchanged recall.Precision rises from 0.050 to 0.404 and SHD falls from 25.1 to 4.6; the layer leads at both sample sizes and improves both wrapped engines.Source: §5.4 · p. 7

6. Conclusion

LUCID routes time series to a deconfounding strategy matched to the inferred confounding regime, then can wrap different discovery engines. Across the out-of-distribution benchmark, it improves directed, lag-resolved graph recovery, though spectral deconfounding and validation scope impose limits.

6. Conclusion

  • LUCID infers the confounding regime with a Marčenko–Pastur router, then applies regime-matched correction and recovers lag-0 structure on the pervasive branch.Separating this layer from causal search lets it wrap different discovery engines.Source: §6 · pp. 7–8
  • Family-weighted directed, lag-resolved F1 is 0.596 for LUCID versus 0.404 for the strongest unwrapped baseline across ten out-of-distribution confounding families.The reported gains also persist under transient and heavy-tailed confounding and in a market-calibrated semi-synthetic study.Source: §6 · p. 8
  • LUCID's gains extend to PCMCI+ and NTS-NOTEARS when the layer is wrapped around those engines.The conclusion characterizes adaptive deconfounding as a useful layer around discovery rather than a replacement for it.Source: §6 · p. 8
  • Validation remains synthetic or semi-synthetic; even the CDS study scores simulated series and edges, leaving real-world causal-ground-truth validation for future work.Spectral trimming can also attenuate genuine lag-0 signal when it overlaps factor-dominated directions.Source: §6 · p. 8

Impact Statement

The impact statement cautions that causal interpretation depends on assumptions about confounding structure and analysis lag. The appendix describes fixed evaluation settings and ten benchmark families spanning sparse and pervasive confounding.

Impact Statement

  • Treat discovered edges as hypotheses to corroborate, because violations of LUCID's confounding-structure or lag-horizon assumptions may leave a graph that no longer represents causal structure.The statement especially cautions against consequential causal claims without domain knowledge, sensitivity analysis, or interventional evidence.Source: Impact Statement · pp. 8–9

A. Additional Method and Experimental Details

  • The appendix specifies hyperparameters, benchmark families, implementation details, and supporting ablations, with settings fixed across the evaluation suite unless stated otherwise.The persistence width τg is the only empirically selected hyperparameter, chosen using generators excluded from the OOD benchmark.Source: A. Additional Method and Experimental… · p. 11, Table 4 · p. 11

A.1. Out-of-distribution family definitions

  • The OOD benchmark weights ten confounding families equally: three use sparse local latent forks, while seven vary properties of pervasive latent processes.The pervasive families vary volatility, loading structure, GARCH behavior, loading density, lag structure, loading magnitudes, and factor persistence.Source: §A.1 · p. 11

A.2. Pair-level evaluation

Pair-level scoring credits an ordered edge at any lag and narrows LUCID's margin over unwrapped baselines. The appendix also reports runtime, CDS simulation design, spectral factor estimation, lag-0 orientation, and the limited role of PDS filtering.

A.2. Pair-level evaluation

  • The lag-collapsed margin over the strongest unwrapped baseline is +0.16, compared with +0.19 when the predicted lag must also be correct.LUCID's pair-level score rises from 0.596 lag-resolved to 0.614, while unwrapped baselines gain 0.043–0.082.Source: §A.2 · pp. 11–12

A.3. Computational cost

  • The LUCID layer alone has median runtime 0.6–0.7 seconds and maximum runtime 1.8–3.5 seconds across three engine configurations.Its median cost on the dense stress case is 1.2 seconds; the B = 200 edge-free-null draws account for most of the overhead.Source: §A.3 · p. 12

A.4. CDS calibrate-then-simulate procedure

  • The CDS-calibrated study fits factor and idiosyncratic-component models to a real market panel, then simulates panels with injected lag-1 edges for evaluation.The procedure VAR(1)-residualizes observed series, extracts factors by PCA, and fits GARCH(1,1)-t and AR(1)+noise models.Source: §A.4 · p. 13

A.5. Spectral factor-count estimation

  • The spectral factor-count estimate uses the Marčenko–Pastur upper edge as a reference and determines how many leading singular directions S−L trims on the pervasive branch.A 1.02 margin avoids counting eigenvalues immediately above the estimated bulk edge, and the trim count is floored at one.Source: §A.5 · p. 13

A.6. Lag-0 orientation

  • Lag-0 candidates are oriented using lead-lag asymmetry motivated by persistence, but the rule is heuristic and may weaken without sufficient temporal persistence.The motivation is a persistent structural time-series model in which an instantaneous effect can propagate through target dynamics, creating lag-1 asymmetry even with Gaussian innovations.Source: §A.6 · p. 13

A.7. PDS edge filter

  • PDS is a conservative post-selection filter over observed predictors, not a general procedure for identifying latent confounding.Its behavior under genuinely unblocked local forks remains an empirical question.Source: §A.7 · p. 14

A.8. Lag-0 recovery ablation

The lag-0 ablation tests whether S−L can recover genuine contemporaneous edges without retaining the factor footprint. The lagged-edge ablation separately tests whether volatility filtering helps on the pervasive branch.

A.8. Lag-0 recovery ablation

  • The ablation compares blanket lag-0 removal, keeping all unresolved contemporaneous edges, and S−L recovery after injecting genuine directed lag-0 edges.Injected edges span factor groups, avoiding the more ambiguous case where a direct edge and its shared factor affect exactly the same group.Source: §A.8 · p. 14
  • 0.980 and 0.948 lag-0 recall, with overall F1 of 0.776 and 0.719, make S−L best at both contamination levels.The comparison averages across five factor processes; lag-0 recall is computed only over injected contemporaneous edges.Source: Table 8 · p. 14, §A.8 · p. 14
  • S−L can recover edges absent from the original skeleton because it constructs candidates from the deconfounded precision matrix, while keep-all is limited to skeleton-proposed edges.Source: §A.8 · p. 14
  • Table 8 is a mechanism/development ablation, not held-out evidence for the persistence width, because these generators helped select it.The ten-family OOD benchmark and Appendix B.2 sensitivity analysis provide the corresponding out-of-sample evaluation.Source: §A.8 · p. 14

A.9. Lag-ℓ≥1 filter ablation

  • 0.592 to 0.451 family-weighted F1 on the full ten-family suite when pervasive-branch lagged edges are filtered; the seven pervasive families fall from 0.528 to 0.323.The shipped method therefore leaves lags ℓ≥1 unchanged on the pervasive branch.Source: Table 9 · p. 14, §A.9 · p. 14

A.10. Robustness to transient confounding

The transient-confounding test varies a latent factor’s active duty cycle using a two-state Markov process. LUCID leads at four of five settings, but the routed tetrad configuration wins at one.

A.10. Robustness to transient confounding

  • The test modulates a latent factor with a two-state Markov process across duty cycles from 1.0 to 0.05.Each dataset has d = 15 and T = 1500, with 20 seeds per duty cycle.Source: §A.10 · p. 15
  • At duty 0.25, routed tetrad performs best, with F1 of 0.715 versus 0.599 for LUCID.The results do not show uniform dominance or systematic degradation as confounding becomes more intermittent.Source: §A.10 · p. 15, Table 10 · p. 15
  • LUCID has the highest F1 at four of five duty cycles and rises from 0.623 at duty 1.0 to 0.818 at duty 0.05.The comparison uses directed, lag-resolved F1 across 20 seeds.Source: Table 10 · p. 15, §A.10 · p. 15

A.11. Robustness to the noise distribution

The noise-distribution test holds one pervasive-factor generator fixed while varying idiosyncratic noise. LUCID leads across all eight conditions, with a larger score drop under Student-t3 noise than Gaussian noise.

A.11. Robustness to the noise distribution

  • The test varies idiosyncratic noise across eight models while holding the pervasive-factor generator fixed; communality c = 0.3 keeps the noise distribution visible in the observations.The models include Gaussian, Laplace, Student-t with ν = 3, uniform, exponential, heteroscedastic location-scale, ARCH-type, and post-nonlinear noise.Source: §A.11 · p. 15
  • LUCID has the highest directed, lag-resolved F1 in all eight noise conditions, with margins of 0.078–0.144 over the strongest non-LUCID baseline.The comparison uses 50 seeds on a fixed pervasive-factor generator with communality c = 0.3.Source: Table 11 · p. 15, §A.11 · p. 15
  • Under Student-t3 noise, LUCID’s F1 falls from 0.489 in the Gaussian condition to 0.438.This is its largest degradation across the tested noise conditions.Source: §A.11 · p. 15

B.1. Router diagnostics and sensitivity

The router diagnostics show that the default favors detecting pervasive cases over minimizing sparse-case misroutes, while the tested router and downstream settings have distinct sensitivity tradeoffs. The collider check also evaluates the persistence gate where its target error is common.

B.1. Router diagnostics and sensitivity

  • Weak factors approaching the noise bulk are a documented limitation; the main OOD benchmark uses the stronger loading range (0.5, 1.5).Source: §B.1 · p. 16
  • At γ = 1.3, the router detects all pervasive cases (FNR = 0) and misroutes 10/30 sparse cases (FPR = 0.333).The diagnostic battery contains 30 sparse-regime and 25 pervasive-regime datasets; γ was not retuned on this battery.Source: Table 12 · p. 16, §B.1 · p. 16
  • Within the tested grid, γ = 2.2 brings FPR to 0 while FNR remains 0, so γ = 1.3 is not the balanced-accuracy-maximizing choice on this battery.The smaller default was fixed independently of the battery and preserves sensitivity to weaker factors.Source: Table 12 · p. 16, §B.1 · p. 16
  • At benchmark loading, FNR is 0 for every tested k, while FPR falls from 0.40 at k = 1 to 0.13 at k = 4.At weaker loading range (0.2, 0.4), k = 4 begins to miss pervasive structure while k ≤ 3 still detects it.Source: §B.1 · p. 16, Table 13 · p. 17

B.2. Sensitivity to fixed constants

  • The downstream sensitivity sweeps vary the sparse-branch PDS level, the S−L edge-free-null level, and the persistence-gate width while reusing the cached base skeleton.The router margin γ is studied separately.Source: §B.2 · p. 17
  • On the collider generator, the shipped gate gives lag-0 skeleton F1 = 1.000 versus 0.957 without it, and admits 0.00 versus 0.75 co-parent false edges per dataset.This is a mechanism check, not held-out validation, because the generator helped select τg.Source: §B.2 · p. 17
  • PDS-sweep family-weighted F1 stays at 0.588–0.598 from 10^-12 through 10^-3, then is 0.585 at 10^-2 and 0.573 at 0.05.At α = 0.05, the layer remains above the 0.411 strongest unwrapped baseline; varying the S−L null level from 0.01 to 0.2 changes F1 by only 0.002.Source: §B.2 · p. 17, Table 14 · p. 18
  • Sweeping τg from 0 to 0.40 changes family-weighted F1 from 0.596 to 0.589, with 0.592 at the default τg = 0.15.This is a robustness check rather than evidence for the gate’s benefit; collider-induced moralization is uncommon in the broad OOD suite.Source: §B.2 · p. 17, Table 14 · p. 18
Loading 2609.31315v1…