PaperlayerView exact revision on arXiv

Source-linked AI summary

Inferring Causal Relations between Two Sequences of Events with Language Models

Nishchal Prasad, Eric Gaussier, Emilie Devijver, Alexander Obeid Guzman, Armen Aghasaryan, Gregor Gössler

arXiv:2609.39406v1cs.AI

TL;DR

Inferring causal direction from just one observation of each event sequence remains a challenge because existing methods often rely on repeated observations. This paper compares sequence-generation likelihoods using pretrained LLMs, and reports strong synthetic results alongside outperformance of traditional baselines on real-world IT-monitoring data.

  • Problem

    Causal direction between two event sequences is difficult to infer from one observation of each, while existing methods generally assume repeated observations or population data.

    Source: §2 · p. 3

  • Method

    The framework uses pretrained LLMs as probability estimators and compares negative log-likelihoods under causal factorizations in both directions, with random encodings and sequence replication for robustness.

    Source: §3 · p. 3, §6 · p. 10

  • Results

    92% average μ-F1 on synthetic data; on real-world IT-monitoring data, the method identifies correct directions and significantly outperforms Granger Causality, VARLiNGAM, and TiMINo.

    Source: §6 · p. 10

  • Takeaways & Limitations

    The approach offers a way to infer direction from a single sequence pair without relying on additional population observations.

    Source: §2 · p. 3, §6 · p. 10

    The method assumes a direct causal relation and always selects a direction, so distinguishing direct causation from independence, confounding, or indirect association requires prior knowledge or another inference step.

    Source: §7 · p. 11

Abstract

from arXiv · show

Causal AI is a branch of Artificial Intelligence which helps understand and reason about cause and effect relationships, not just patterns or correlations. Causal discovery aims to infer elements of the underlying causal structure--often represented as a directed graph--from observational and, when available, interventional data. While causal discovery is the fundamental step for moving beyond mere associations toward genuine understanding, and thus the basic building block of causal AI, it becomes intrinsically difficult when causal relations must be inferred from single observations. In such situations, standard causal discovery methods cannot be used and one has to identify causal relations from limited amount of information. This is typically the case for, e.g., sequences of events produced by different alarms which need to be analyzed on the fly to detect abnormal phenomena, which are usually rare. We show in this study that it is possible to leverage the predictive power of Large Language Models (LLMs) to infer causal relations between only two sequences of events. This approach, which is validated on both synthetic and real data, provides better results than standard causal discovery algorithms on several time series data, even though these data were converted into smaller, single observed sequences.

1 Introduction

The paper addresses causal-direction inference from a single observation of each event sequence, where methods relying on repeated observations cannot be applied. It proposes comparing sequence-generation likelihoods with pretrained language models and evaluates the approach on synthetic and real-world data.

1 Introduction

  • With only one trace per process, existing causal-direction methods that require repeated observations cannot estimate distributions, fit models, or aggregate event intensities.The paper frames this as a setting with no population from which additional samples can be drawn.Source: §1 · p. 1
  • The framework compares sequence likelihoods under competing causal directions, treating the higher-probability factorization as evidence for the generating direction.Pretrained LLMs are used as zero-shot probability density estimators rather than semantic knowledge bases.Source: §1 · pp. 1–2
  • Random encoding mapping mitigates token semantic bias, while sequence replication strengthens in-context learning for out-of-distribution event sequences.The evaluation varies sequence lengths, encoding configurations, and LLM backbones, and also tests real-world IT-monitoring data.Source: §1 · p. 2
  • The authors present this as the first method to orient causation between two event sequences from a single observation, using LLMs as structural probability estimators.The introduction also situates causal direction as important for distinguishing root-cause diagnosis from mere correlation [43] [52].Source: §1 · pp. 1–2

2 Related Work

Prior causal-direction methods use statistical or algorithmic asymmetries, but many require repeated observations or sufficiently many event recurrences. The paper identifies a practical gap for single observations and contrasts its structural use of LLMs with approaches based on semantic knowledge.

2 Related Work

  • For continuous data, prior approaches exploit functional-model asymmetries, independence of mechanism, or learned classification from distributions, generally using multiple i.i.d. observations.Examples include LiNGAM, additive-noise and post-nonlinear models [44, 20, 51], independence-based methods [23, 15, 14, 32–34], and learned classifiers [30] [31] [47].Source: §2 · p. 2
  • Event-sequence methods commonly need many sequence pairs or event recurrences to estimate intensities, conditional entropies, or compression models [42] [45] [1] [22] [40] [48] [53].Logic-based frameworks also aggregate evidence across many event traces.Source: §2 · pp. 2–3
  • For a single observation, the paper reports no prior method for causal direction without extra population data; existing fitted compressors need recurrences, leaving a practical instantiation gap for the algorithmic Markov condition.The condition establishes in-principle single-object inference by comparing Kolmogorov complexities of the two factorizations, but does not provide a computable event-sequence method.Source: §2 · p. 3
  • Earlier LLM causal-discovery approaches rely on semantic variable names or causal relations extracted from text [29], whereas this work uses LLMs to model non-linguistic sequence patterns [18] [35] [11] [38].The cited sequence-modeling work includes out-of-distribution time-series prediction and pattern completion with randomly drawn vocabulary tokens [18] [35].Source: §2 · p. 3

3 Methodology

The methodology formalizes causal-direction inference from one observed pair of event sequences and uses LLM-estimated sequence probabilities to compare causal hypotheses. Random encodings and sequence replication address token bias and short-sequence context limitations. [p21, p35, p49, p66]

3 Methodology

  • Given one observation of each sequence and a direct causal relation, the task is to determine which hidden process causes the other.The framework assumes separate cause-generating and effect-given-cause mechanisms.Source: §3 · p. 3

3.1 LLM-based estimation

  • LLMs estimate autoregressive sequence likelihoods, and the Sequence Generation Rule selects the direction with the smaller estimated causal score.The scores are estimated over alternative encodings; the method uses Eq. 2 because the two estimation approaches yield very similar results in practice.Source: §3.1 · p. 4

3.2.1 Encodings.

  • Random bijective token mappings reduce dependence on any one encoding, but the full encoding space is too large to explore exhaustively.The method samples mappings from subsets of a universal vocabulary.Source: §3.2.1 · p. 4

3.2.2 Replication.

  • Because LLMs are left-to-right and short sequences can hinder in-context learning, replication appends a copy to expose the original sequence as context.The authors also describe replication as increasing latent-pattern frequency and induction-head activation.Source: §3.2.2 · pp. 4–5

3.3 LLM-based Causal Discovery Algorithm

  • The overall algorithm combines the Sequence Generation Rule with random encodings and context replication to infer direction from observed sequence pairs.For each encoding, it accumulates scores under both orientation hypotheses and averages them across encodings.Source: §3.3 · p. 5
  • Averaging causal scores across K encodings outperformed max-voting, which retains token-biased decisions instead of smoothing score magnitudes.The method evaluates both causal hypotheses across encodings, averages the scores, then applies the lower-score rule.Source: §3.1 · p. 4, §3.3 · p. 5

4 Experimental Setup

The experimental setup section introduces the method’s hyperparameters, test data, and baseline methods. [p89]

  • The experimental setup covers hyperparameters, test data, and baseline methods.Source: §4 · p. 5

4.1 Hyper-parameters

The hyperparameter experiments vary language-model size, encoding count, and replication, using synthetic data to choose settings for the real dataset. [p08, p09, p10, p11, p12]

4.1 Hyper-parameters

  • The experiments primarily use Llama-3.2-3B, with GPT-2 (about 124M parameters) and Llama-3.2-1B as smaller-model comparisons.Experiments use one A100/H100 GPU; Llama variants are loaded in 16-bit precision to reduce GPU memory use.Source: §4.1 · pp. 5–6
  • Encoding counts are K∈{0, 1, 51, 311, 1111}; K=0 uses the observed sequences without encoding.The nonzero counts were chosen randomly as increasing odd integers.Source: §4.1 · p. 6
  • Replication experiments compare r=0 with r=1 to test whether pseudo-bidirectional context helps, especially for sequences of 32 symbols or fewer.Source: §4.1 · p. 6
  • A synthetic-data ablation selects r and K for the real-data experiments, using a universal symbol set of uppercase and lowercase letters and decimal digits.Source: §4.1 · p. 6

4.2 Data

The evaluation uses one synthetic dataset and four real-world time-series datasets converted into symbolic events. Synthetic pairs have a known one-way generating process; real-world evaluation uses direct causal pairs.

4.2 Data

  • Five datasets are evaluated: one synthetic and four real-world time series transformed into symbolic event sequences.The sequences are treated as single observations, and their event symbols carry no semantic meaning.Source: §4.2 · p. 6

4.2.1 Synthetic data.

  • Synthetic pairs are generated by a DFA producing X and a DFT mapping X to Y, establishing a one-way causal dependency.Distinct DFA/DFT pairs generate each evaluated pair; symbols are randomly chosen from lowercase letters and digits.Source: §4.2.1 · p. 6, Table 1 · p. 6

4.2.2 Real data.

  • The real IT-monitoring data cover MoM, Storm Ingestion, Web Activity, and Antivirus Activity; evaluation uses direct causal edges and excludes indirect pairs.The source provides ground-truth causal graphs, and indirect pairs are excluded because hidden mediators make direction recovery harder.Source: §4.2.2 · p. 6
  • Real-valued series are discretized into three quantile states, with changes between states treated as events, yielding up to six transition types per variable.MoM is first-order differenced, smoothed, and standardized; the stationary subsets need no preliminary processing.Source: §4.2.2 · p. 6

4.2.3 Pre-processing event sequences.

  • Whitespace separates event symbols so each is tokenized independently; real-pair evaluations also balance direction labels through five forward and five reversed assignments.Antivirus Activity and Web Activity use sequences about one-third the total length because the extracted sequences were too long.Source: §4.2.3 · p. 7, §4.2.2 · p. 7

4.3 Baseline methods

The baselines include standard zero-shot LLM prompting and classical time-series causal discovery on real data. Classical methods use more data than the proposed single-pair approach, and their unoriented outputs are handled explicitly.

4.3.1 Zero-shot LLM prompting baselines (synthetic & real data).

  • Zero-shot baselines query the same backbone LLMs, with additional larger models, using unencoded, randomly encoded, or encoded-and-replicated sequences.Max-voting over 51 or 311 encodings is also tested for encoded prompting variants.Source: §4.3.1 · p. 7
  • On synthetic data, Table 3 compares standard prompting variants with the proposed method across sequence lengths and models.The table reports average test μ-F1 and includes the method with K=1111 and r=1.Source: Table 3 · p. 7

4.3.2 Classical causal discovery baselines (real data only).

  • Real-data baselines include Granger Causality, VARLiNGAM, and TiMINo; except Granger, they use full numerical series and more information than the single discretized pair.Non-oriented outputs are randomly assigned a direction with p=0.5 and averaged over 10 trials; binary-classification metrics are equivalent here.Source: §4.3.2 · p. 7, §4.3.3 · p. 8

5 Experimental results

The experiments evaluate sequence-generation scoring on synthetic data and test its strongest configuration on unseen real IT-monitoring data. Performance depends on sequence length, replication, encoding count, and backbone capability.

5.1.1 General conclusion.

  • 91.94% μ-F1 is achieved by the Sequence Generation Rule on synthetic data, while standard zero-shot prompting averages near-random 50% μ-F1.Zero-shot variants using encoded sequences also remain near random, with μ-F1 between 0.4 and 0.6.Source: §5.1.1 · p. 8

5.1.2 Effect of sequence lengths.

  • With replication, μ-F1 for lengths 32–83 rises from 0.5153 to 0.9322, while performance still ranges from 0.7888 on lengths 16–41 to 0.9900 on lengths 128–166.Without replication, lengths 16–41 and 32–83 remain near random, while lengths 128–332 exceed 0.94 μ-F1.Source: §5.1.2 · p. 8

5.1.3 Effect of replications.

  • Across lengths at K=1111, replication raises average μ-F1 from 0.7477 (r=0) to 0.9194 (r=1), with the clearest gains on shorter sequences.Replication improves sequence prediction by giving the language model more context.Source: §5.1.3 · p. 8

5.1.4 Effect of encodings.

  • With replication, increasing K from 51 to 1111 raises μ-F1 from 0.8237 to 0.9322 for lengths 32–83; larger encoding pools also make performance more stable.Raw sequences and single random encodings are unstable; without replication, increasing K does not resolve short-sequence performance near 0.50–0.58 μ-F1.Source: §5.1.4 · pp. 8–9

5.1.5 Impact of the backbone LLM.

  • Under r=1 and K=1111, GPT-2 peaks at 0.7706 μ-F1, whereas Llama-3.2-1B and Llama-3.2-3B reach 0.9066 and 0.9194, respectively.Replication and larger encoding pools improve performance across backbones, but results depend on the underlying language model capability.Source: §5.1.5 · p. 10

5.2 Test results on real data

  • The real-data evaluation applies the synthetic-data configuration to an unseen test set without hyperparameter tuning.The tested configuration is Llama-3.2-3B with K=1111 and r=1.Source: §5.2 · p. 10
  • On real data, the method outperforms classical and zero-shot baselines on most datasets, leading Antivirus Activity (0.813), MoM (0.700), and Web Activity (0.714).On Storm Ingestion it scores 0.556, close to Granger Causality at 0.558 and above TiMINo at 0.552.Source: §5.2 · p. 10, Table 6 · p. 10

6 Conclusion

The paper applies pre-trained LLMs as training-free probability estimators to orient causal direction between event sequences from one observation. Synthetic and real-data results support the approach, with random encodings and sequence replication improving robustness in practice.

6 Conclusion

  • The framework compares negative log-likelihoods for the two causal orientations, using LLM sequence modeling rather than semantic knowledge.Random bijective encodings reduce semantic bias, while sequence replication strengthens in-context learning.Source: §6 · p. 10
  • The method achieves an average μ-F1 of 92% on synthetic data and outperforms traditional baselines on real-world IT-monitoring data.The real-data comparison includes Granger Causality, VARLiNGAM, and TiMINo, despite their access to full continuous time series.Source: §6 · p. 10
  • Ablation studies highlight random encodings and replication as important for performance stability.Averaging across randomized encodings balances variation in the resulting predictions.Source: §6 · pp. 10–11
  • The findings suggest pre-trained sequence models can provide structural signals for orienting causal relations from a single observation; future work will add temporal information and extend the framework to full causal graphs.The stated application settings include web-service logs, telecommunications, and monitoring data.Source: §6 · p. 11

7 Limitations

The method’s scope is constrained by an assumed direct link, raw-probability model choice, finite context and computation, noisy real-data discretization, and evaluation without semantic metadata.

7 Limitations

  • The rule assumes a direct causal link and always orients the pair, so it cannot distinguish direct causation from independence, confounding, or indirect association.Whether a direct link exists must be established through prior knowledge or another inference step.Source: §7 · p. 11
  • The algorithm requires raw probability estimates, so the experiments use base Llama-3.2 models rather than instruction-tuned variants.The excluded variants are Llama-3.2-3B-Instruct and Llama-3.2-1B-Instruct.Source: §7 · p. 11
  • Input length is bounded by the LLM context limit, while quadratic attention cost leads the authors to use a moderately sized model and truncate very long event sequences.For GPT-2, the cited context limit is 1,024 tokens; inference cost also grows with the number of encodings.Source: §7 · p. 11
  • Real-data discretization can introduce spurious state changes and miss subtle events, obscuring causal dependencies and contributing to lower μ-F1 than on synthetic data.The reported real-data scores remain below the > 0.90 levels achieved on synthetic data.Source: §7 · p. 11
  • The evaluation treats event symbols as non-semantic, leaving the potential additional evidence from event names, system topology, or domain knowledge outside its scope.Combining such metadata with the method remains unevaluated.Source: §7 · p. 11
Loading 2609.39406v1…