Source-linked AI summary
Large Causal Models for Temporal Causal Discovery
Nikolaos Kougioulis, Nikolaos Gkorgkolis, MingXue Wang, Bora Caglayan, Dario Simionato, Andrea Tonon, Ioannis Tsamardinos
TL;DR
Temporal causal discovery is limited by dataset-specific methods, poor scaling, narrow synthetic training distributions, and scarce diverse datasets with ground-truth graphs. The paper develops LCMs through heterogeneous supervised pretraining and evaluates them across diverse temporal settings. LCMs match or outperform established baselines while providing fast, single-pass graph prediction, although their conclusions remain bounded by causal assumptions.
Problem
Temporal causal discovery methods face poor dimensional scaling, limited zero-shot generalization, narrow synthetic training, and scarce diverse datasets with ground-truth causal graphs.
Method
LCMs are pretrained foundation models for temporal causal discovery using a large heterogeneous corpus and supervised learning across diverse data distributions.
Results
LCMs consistently match or outperform established classical baselines across in-distribution and out-of-distribution settings while enabling fast, single-pass graph prediction.
Takeaways & Limitations
LCMs provide a foundation-model-style alternative to fitting a new causal-discovery model for each dataset.
Takeaways & Limitations
LCMs assume causal sufficiency, no contemporaneous effects, causal stationarity, and additive noise; violated assumptions may make predictions reflect training-distribution biases.
Abstract
from arXiv · showhide
Causal discovery for both cross-sectional and temporal data has traditionally followed a dataset-specific paradigm, where a new model is fitted for each individual dataset. Such an approach limits the potential of multi-dataset pretraining. The concept of large causal models (LCMs) envisions a class of pre-trained neural architectures specifically designed for temporal causal discovery. Prior approaches are constrained to small variable counts, degrade with larger inputs, and rely heavily on synthetic data, limiting generalization. We propose a principled framework for LCMs, combining diverse synthetic generators with realistic time-series datasets, allowing learning at scale. Extensive experiments on synthetic, semi-synthetic and realistic benchmarks show that LCMs scale effectively to higher variable counts and deeper architectures while maintaining strong performance. Trained models achieve competitive or superior accuracy compared to classical and neural baselines, particularly in out-of-distribution settings, while enabling fast, single-pass inference. Results demonstrate LCMs as a promising foundation-model paradigm for temporal causal discovery. Experiments and model weights are available at https://github.com/kougioulis/LCM/.
1 Introduction
Temporal causal discovery seeks causal structure in time series, but classical and neural methods face scaling, assumption, generalization, and data-diversity limits. The paper motivates LCMs as pretrained models for heterogeneous temporal discovery and reports scalable, efficient performance.
- Classical temporal CD methods rely on conditional independence testing or combinatorial search, whose complexity scales poorly with input dimensionality.
- Neural temporal CD methods remain mostly dataset-specific and struggle to generalize across heterogeneous systems, especially in zero-shot settings.
- Prior LCM work was limited to low dimensions and narrow synthetic training distributions, while scarce diverse datasets with ground-truth graphs constrain generalization.
- The paper formalizes LCMs as foundation models for temporal CD, replacing dataset-specific algorithms with foundation-model-style pretraining across heterogeneous distributions.
- LCMs are presented as achieving competitive or superior performance to classical methods while reducing runtime through single-pass causal discovery.
2 Related Work
Related work includes task-specific neural causal-discovery methods and an earlier proof-of-concept LCM. This paper positions its approach around heterogeneous pretraining, scalability, and zero-shot evaluation.
- Most pretrained causal approaches remain task- or dataset-specific and lack the broad generalization associated with foundation models.
- Earlier neural methods commonly use limited synthetic datasets, output summary graphs, and do not demonstrate large-scale zero-shot generalization in temporal heterogeneous settings.
- Prior LCM work was a proof of concept limited to small systems, narrow synthetic distributions, and sharply degrading performance with increasing dimensionality.
- The paper emphasizes data diversity, heterogeneous training, and zero-shot or out-of-distribution evaluation as prerequisites for scalable temporal causal discovery.
- The proposed LCM infers lagged causal graphs from multivariate time series and learns a parametric mapping rather than performing dataset-specific DAG search.
4 Model Overview
The LCM combines convolutional temporal embeddings, Transformer encoding, correlation-based training aids, and a feedforward graph-prediction head. Its supervised objective combines edge prediction with correlation regularization.
- The implemented LCM uses a convolution-enhanced Transformer encoder for multivariate time-series causal discovery.
- Architecture: Inputs are normalized, noise-padded to model limits, projected with Conv1D layers, and augmented with positional encodings.
- Architecture: Self-attention, feedforward layers, residual connections, and Conv1D distillation produce contextualized representations for long-horizon processing.
- Architecture: Lagged cross-correlations are concatenated as training aids, while a sigmoid feedforward head outputs probabilities in a lagged adjacency tensor.
- Objective: The composite loss combines supervised binary edge prediction with a correlation-based regularizer weighted by λedge = 1 and λcorr = 3/4.
5 Training Setup
Training uses paired time-series and causal-graph samples from synthetic, realistic, and mixed collections. The main corpus contains 275k instances and supports both in-distribution and out-of-distribution evaluation.
- LCM supervised training requires paired samples and combines synthetic, semi-synthetic, and realistic datasets to promote zero-shot generalization under distribution shift.
- Synthetic data: The primary Synthetic_2 collection contains 230k instances generated across variable counts, lags, graph densities, and linear or nonlinear mechanisms.
- Evaluation: Mechanistic fMRI and Kuramoto simulators are reserved exclusively for out-of-distribution evaluation and are never used for training.
- Realistic data: Realistic causal twins span energy, weather, and transportation domains and yield 45k instances.
- Training corpus: The main LS corpus combines synthetic and realistic data into 275k instances and 137.5M time points, with holdout and separate OOD benchmarks.
6 Experimental Setup
The experimental setup evaluates LCM variants across increasing model capacities and compares them with established temporal causal discovery methods using lagged-graph AUC and corrected paired significance tests.
- LCM variants range from approximately 900K to 24M parameters to study scalability with variable count and data size.Smaller models support ablations and comparisons, while larger models examine scaling behavior.
- Edge discovery is measured by AUC from the discovered lagged adjacency tensor, with higher values indicating better ground-truth graph recovery.Edges are ranked by confidence, and the area under the TPR–FPR curve is computed across thresholds.
- Standard errors are estimated across datasets within each collection.
- AUC differences are tested with the paired Wilcoxon signed-rank test and Bonferroni correction across multiple comparisons.The correction uses αcorrected = α/k, where k is the number of comparisons.
- LCMs are compared with PCMCI, DYNOTEARS, and VARLinGAM, representing constraint-based, score-based, and functional-model approaches.A pre-trained Transformer is additionally included for small datasets with V = 5 and ℓmax = 3.
- Baseline outputs are converted to lagged adjacency tensors for fair comparison, using inverse p-values or bootstrap-estimated edge probabilities as confidence measures.The confidence construction depends on the type of baseline method.
7 Experimental Results
Across in-distribution, out-of-distribution, and runtime evaluations, LCMs maintain strong causal-discovery performance while scaling beyond prior small-system settings. Diverse training data and realistic-data mixtures improve generalization, and single-pass inference provides computational advantages over conventional methods.
- Overall evaluation: LCMs consistently match or outperform classical baselines across in-distribution and out-of-distribution temporal causal-discovery settings.The evaluation covers holdout, out-of-sample, semi-synthetic, and realistic benchmarks.
- Observed statistics: Training aids substantially improve AUC, with CI improving over the baseline and CR providing further gains across tested correlation weights.The highest mean AUC occurs for λcorr ∈ [0.25, 0.75], and λcorr = 0.75 is selected subsequently.
- Synthetic-realistic mixtures: Mixed synthetic-realistic training consistently outperforms synthetic-only training on semi-synthetic fMRI and real-data–derived Power and Climate benchmarks.These evaluations measure out-of-distribution performance.
- Synthetic-realistic mixtures: 20% realistic data yields systematic, statistically significant gains, while higher proportions provide marginal or inconsistent improvements.The adopted 80/20 synthetic-to-realistic mixture balances structural diversity and real-world fidelity.
- Scaling and synthetic performance: LCMs achieve state-of-the-art AUC on synthetic benchmarks and scale beyond previously studied 3–5-variable systems.The authors attribute prior scaling failures to restrictive training distributions rather than architectural constraints.
- Runtime: LCM runtimes remain effectively independent of input dimensionality, unlike traditional methods whose costs scale superlinearly with variable count and temporal lag.LCM inference uses a single forward pass instead of fitting a new model for each dataset.
8 Model Complexity
The LCM parameterization decomposes across embeddings, encoder blocks, optional distillation layers, and the causal prediction head. Complexity is driven by pairwise causal-edge terms, while runtime advantages arise from a constant-time forward pass.
- The parameter-count expression uses B, dmodel, nheads, dff, Itrain, Idistil, and convolutional kernel size k to represent architecture and training-aid choices.
- The trainable-parameter count includes input embeddings, B encoder blocks, optional (B −1) distillation layers, and the final causal prediction head.
- Transformer-based LCMs remain effectively independent of input dimensionality in runtime, whereas PCMCI and DYNOTEARS scale superlinearly with lag and variable count.
- Θ(V 2 maxℓmax) complexity is dominated by pairwise causal edges in the feedforward head.
- For Vmax = 25 and ℓmax = 3, the head contributes ≈2M parameters and the total reaches ≈75M with 24M hyperparameters.
9 Training Data–Model Size Convergence
Experiments varying model size and training-data volume show that capacity and data availability jointly constrain LCM performance. Smaller models saturate earlier, whereas larger models continue benefiting from additional data.
- The study trains 500K, 1M, and 2M parameter LCMs on datasets containing 10K, 25K, 50K, or 100K samples.Experiments use five random seeds and fixed validation/test sets of 2 · 10^3 samples.
- Smaller models saturate early with diminishing returns from additional data, while larger models continue improving as training data increases.
- Scaling model capacity or data alone yields limited gains, whereas scaling both together leads to consistent improvements.
- The results inform practical design choices for large-scale temporal causal discovery rather than establishing a universal scaling law.
10 Concluding Remarks & Future Work
LCMs amortize temporal causal discovery through supervised pretraining, producing fast single-pass graph predictions without optimization. They show strong in- and out-of-distribution generalization, while future work targets broader causal settings and scales.
- LCMs are foundation models for temporal causal discovery that amortize supervised graph prediction across mixed data collections.
- Single-pass inference produces causal graphs without optimization, enabling fast prediction.
- LCMs consistently match or outperform established baselines on in- and out-of-distribution settings.
- Future work includes latent confounders, contemporaneous effects, higher-dimensional systems, and longer horizons.
11 Limitations
LCMs rely on the causal assumptions specified by the paper and do not detect or correct violations. Under violated assumptions, predictions may reflect training-distribution biases rather than true causal structure.
- LCMs do not detect or correct violations of the causal assumptions defined in Section 1.
- When assumptions are violated, predictions may reflect training-distribution biases rather than true causal structure.
12 Impact Statement
LCMs enable scalable, zero-shot temporal causal discovery without dataset-specific retraining. The authors suggest this could accelerate scientific discovery by reducing reliance on costly randomized control trials and A/B testing.
- LCMs perform temporal causal discovery without dataset-specific retraining.
- The approach supports scalable, zero-shot temporal causal discovery.
- The authors suggest LCMs could reduce reliance on costly randomized control trials and A/B testing in biology and marketing.
Causal Assumptions & Brief Definitions
The framework assumes additive noise, causal stationarity, no latent confounders, and no contemporaneous effects, while representing temporal dependencies with lagged causal graphs. LCMs standardize heterogeneous inputs through padding, diverse synthetic generation, auxiliary correlations, and multi-label edge prediction.
- Brief Definitions: A lagged causal graph is a DAG whose directed edges represent direct causes between time-shifted variables.
- Brief Definitions: A summary graph retains whether any lagged causal relationship exists while discarding the specific lag or delay.
- Input Handling and Padding: Fixed-dimensional padding supports batch processing across variable-length series, variable counts, and differing maximum lags.
- Synthetic Data Generation: Synthetic generation varies graph size, lag ranges, density, and linear or nonlinear mechanisms, while bounded functions improve stationarity and scalability.