PaperlayerView exact revision on arXiv

Source-linked AI summary

seq2cause: One Autoregressive Backbone, Four Causal Discovery Tasks in Event Sequences

Hugo Math

arXiv:2609.31801v1cs.LG

TL;DR

Discrete event sequences raise causal questions about event-to-event and event-to-outcome relations, across single sequences and populations. Seq2Cause repurposes one pretrained autoregressive model for all four regimes, with excess cross-entropy controlling causal-identification error. It reports four-regime results on nonlinear SCMs and vehicle logs where existing methods are inaccurate or intractable.

  • Problem

    Causal discovery in high-vocabulary single-stream sequences spans event-to-event and event-to-outcome relations at sample and population levels, settings where pair co-occurrences are sparse and existing approaches do not cover the regimes jointly.

    Source: §1 · p. 1, §2 · p. 3

  • Method

    Seq2Cause repurposes pretrained autoregressive conditional probabilities for CMI-based causal discovery across four regimes without task-specific retraining.

    Source: §1 · p. 2

  • Results

    Seq2Cause populates all four regimes with one frozen backbone; it reports 91% F1 on sample-level event graphs where the strongest baseline reaches 69%.

    Source: §1 · p. 3, §7 · p. 10

  • Takeaways & Limitations

    The model’s excess cross-entropy provides a shared, monitorable quantity that bounds causal-identification errors across regimes, so improved next-token prediction tightens the stated guarantees.

    Source: §1 · p. 3, §7 · p. 11

    The guarantees rely on temporal precedence, causal sufficiency, and strong faithfulness, which requires valid causal associations to exceed a significance threshold.

    Source: §5.2 · p. 6

Abstract

from arXiv · show

Complex systems such as vehicles, patients, or genomes emit discrete event sequences whose operative question is causal, not predictive: which events cause which other events, and which cause higher-level outcomes such as failures or diseases? This question decomposes along two axes -- dependency type (event $\to$ event vs.\ event $\to$ outcome) and causal scope (single sequence vs.\ population) -- yielding four structurally distinct regimes with different identifiability conditions. No existing method addresses more than one, because all assume multi-stream structure with low vocabulary, and none scales beyond a few hundred event types. We present \textsc{Seq2Cause}, a unified framework that resolves all four regimes through a single shared primitive: a pretrained autoregressive model repurposed as an amortized conditional independence testing engine requiring no task-specific retraining. We establish a prediction--causality duality: the model's excess cross-entropy simultaneously bounds causal identification error across all four regimes, so that every improvement in next-token prediction tightens causal guarantees for free. On nonlinear SCMs (vocabularies up to $8{,}000$ types) and real-world vehicle diagnostic logs ($29$K event types, $474$ failure outcomes), \textsc{Seq2Cause} is the first method to populate all four regimes at scale with a single frozen backbone. Existing methods are either inapplicable, inaccurate, or computationally intractable in this setting.

1 Introduction

Seq2Cause targets four distinct causal-discovery regimes in discrete event sequences using one pretrained autoregressive backbone. Its central claim is that prediction quality controls causal-identification guarantees across those regimes.

1 Introduction

  • Discrete event sequences arise in vehicles, patient trajectories, and genomes, where the scientific question concerns causal relations among events and outcomes rather than prediction alone.Examples include diagnostic codes preceding failures and ordered regulatory events in genomes.Source: §1 · p. 1
  • The framework crosses event-to-event versus event-to-outcome dependencies with sample-level versus population-level scope, yielding four regimes with distinct identifiability conditions and estimands.For example, asking whether Fire@t1 caused Smoke@t2 differs from asking whether Fire causes Smoke in general.Source: §1 · p. 2, Table 1 · p. 2
  • A pretrained autoregressive model supplies conditional probabilities for CMI testing, enabling batched discovery across all four regimes without task-specific retraining.The framework recovers event graphs and outcome Markov boundaries from a single sequence or an aggregated dataset.Source: §1 · p. 2
  • Seq2Cause is the first method reported to populate all four regimes with one frozen backbone on nonlinear SCMs and vehicle logs, where existing methods are inaccurate or intractable.The evaluation covers vocabularies up to 8,000 types for nonlinear SCMs and 29,100 event types plus 474 outcomes for vehicle logs.Source: §1 · p. 3
  • Excess cross-entropy ε controls CMI estimation error, sample-time graph soundness, and MDL population edge-inclusion error, linking prediction quality to causal guarantees.The method uses the same monitorable quantity across regimes, with no regime-specific calibration.Source: §1 · p. 3

2 Related Work

Prior causal-discovery methods largely assume multiple streams with comparatively small vocabularies, while single-stream, high-vocabulary sequences create different statistical and computational challenges. Existing outcome-boundary and root-cause methods also do not jointly address these settings.

2 Related Work

  • Multi-stream methods scale quadratically or worse with event types |X|; Hawkes estimation has |X|^2 parameters and the family becomes computationally intractable when |X| > 10^3.Source: §2 · p. 3
  • In single-stream sequences, pair co-occurrences become rare and empirical joint distributions degenerate, while one realization provides no long stationary trajectory for population extraction.Sliding-window arguments used for multi-stream methods do not apply to a single vehicle trip, patient visit, or genome sequence.Source: §2 · p. 3
  • Classical Markov-boundary algorithms face |X|^2 candidate pairs and distributional assumptions, and none jointly handles high-vocabulary single-stream X→Y and X→X discovery.Existing root-cause approaches require known topology, continuous multivariate streams, or anomaly scores rather than a causal graph.Source: §2 · p. 3

3 Notations and Background

The background establishes a shared probability-space representation for events and outcomes, then defines the graph objects used to express sample-level causal structure and its type-level summary.

3 Notations and Background

  • The event chain lives on infinite sequences over a finite alphabet, while binary outcome labels are measurable functions of the full trajectory and their head estimates probabilities from event prefixes.This represents event indicators and outcomes within one probability-space framework.Source: §3 · p. 4

3.1 Definitions

  • A Markov Boundary is the minimal conditioning set that makes an outcome independent of all remaining variables.No proper subset of the boundary has this property.Source: §3.1 · p. 4

3.2 Summary Graph

  • The Sample-Summary Graph projects a sample-time graph onto unique event types, including a type-level edge when that edge appears at least once in the time graph.Summary graphs may contain cycles even though time graphs are DAGs under temporal precedence.Source: §3.2 · p. 4, Table 2 · p. 4

4 Background and Problem Formulation

The problem formulation models events as a discrete stochastic process with potentially full-history dependence, alongside binary outcomes generated from event histories. It defines event-to-event and event-to-outcome graph objects under stated structural assumptions.

4 Background and Problem Formulation

  • The event process takes values in a finite discrete alphabet, has no finite-order Markov assumption, and outcomes are binary variables generated conditionally on event history.The event chain is autonomous, and outcomes are assumed to be driven solely by events with no inter-outcome effects.Source: §4 · p. 5
  • The two-chain data-generating process partitions edges into intra-chain event-to-event links and cross-chain event-to-outcome links, with no outcome-to-event or outcome-to-outcome edges.The resulting sample-level graph is a DAG by construction.Source: §4 · p. 5
  • The sample-time graph links event time steps when an earlier realization directly causes a later event, while the sample Markov Boundary Graph links event times to outcomes they cause.The outcome-boundary definition uses the event-to-outcome parents under the stated data-generating process.Source: §4 · p. 5

5 The SEQ2CAUSE Framework

SEQ2CAUSE repurposes autoregressive next-token models as a shared conditional-independence testing engine for event and outcome targets. Its sample-level and population-level procedures use model approximation quality to bound or correct causal identification, under stated assumptions.

5.1 Neural Autoregressive Density Estimation

  • Autoregressive models estimate next-event and outcome distributions from histories, querying outcome likelihood at every step although outcomes are realized only at the end.The models are trained once by cross-entropy minimization.Source: §5.1 · p. 5

5.2 Assumptions and ϵ-Oracle Model

  • The ϵ-oracle assumption bounds identification error by achieved model quality, while strong faithfulness requires valid causal associations to exceed threshold τ.The framework also assumes temporal precedence and causal sufficiency.Source: §5.2 · pp. 5–6

5.3 Sample-Level: The Universal CMI Primitive

  • Conditional mutual information between a target and candidate event, given the full trajectory history, supplies a shared conditional-independence test for event or outcome targets.For lagged event effects, intervening on mediators closes an indirect path; the full-history adjustment relies on causal sufficiency.Source: §5.3 · p. 6
  • The Monte Carlo CMI estimator is consistent, uses two model queries per particle, and has error bounded by model approximation error.The estimator is parallelizable across sequence positions and particles.Source: §5.3 · p. 6, §5.4 · p. 7

5.4 Sample-Level Soundness

  • Under ϵ-strong faithfulness, causal sufficiency, and temporal precedence, sample-level graph recovery is sound as N→∞, with no false positives as ϵ→0.The CMI estimation error is bounded in terms of the model’s binary-target approximation error.Source: §5.4 · p. 7
  • Under the sample-level soundness conditions, the earliest event with a directed path to an outcome’s Markov boundary is identifiable as its root cause.The paper presents this as a root-cause identifiability result without topology assumptions or interventional data, under the listed assumptions.Source: §5.4 · p. 7

5.5 Population-Level Discovery via Minimum Description Length

  • Population discovery selects a type-level graph by MDL, including u→v when accumulated likelihood gain exceeds its encoding cost.The criterion uses a one-time edge encoding cost and a dataset negative log likelihood.Source: §5.5 · p. 7

6 Experiments

The experiments evaluate SEQ2CAUSE across four causal regimes using nonlinear synthetic SCMs and vehicle diagnostic sequences. Results report strong sample-level event-to-event performance, population-level gains with MDL error correction, and robustness to model error and vocabulary growth.

6.1 Setup

  • The evaluation covers nonlinear SCMs and vehicle diagnostic sequences, comparing SEQ2CAUSE with task-specific causal, attribution, and association baselines.The vehicle data comprise 300,000 sequences, 29,100 event types, and 474 outcomes; evaluation uses weighted precision, recall, and F1.Source: §6 · p. 8, §6.1 · pp. 8–9

6.2 Unified Four-Regime Evaluation

  • Sample-level X→X reaches 91% F1 at |X|=1,000, while corrected population-level X→X reaches F1 65.4% (Prec. 81.5%, Rec. 54.6%).For sample-level X→X, Neural Granger is the strongest baseline; for population-level X→X, the corrected method is reported as the only one maintaining both high precision and moderate recall.Source: §6.2 · p. 9
  • At |X|=29,100, all five classical vehicle-sequence baselines exceed three days on 50,000 sequences; at K=500 and n=500, their precision and recall collapse to zero.The passages attribute these outcomes to vocabulary-driven intractability and sparse co-occurrence support, respectively.Source: §6.2 · pp. 9–10

6.3 Generative Error and Scalability

  • At ϵ≈0.1, structural recovery occurs before exact model convergence; as ϵ increases, precision stays near 0.95 while recall degrades.The reported conservative failure mode is missed edges rather than hallucinated ones.Source: §6.3 · p. 10, Figure 2 · p. 10
  • F1 remains stable at ≈81% as vocabulary grows from |X|=10 to 8,000, while the candidate edge space grows exponentially.The passage attributes this scalability to amortized density estimation by the pretrained autoregressive backbone.Source: §6.3 · p. 10
  • Without MDL error correction, precision collapses near zero beyond N≈1,000; with correction, it stays above 80% across n∈[10, 20·10^4].The passages attribute the uncorrected collapse to approximation errors accumulating across sequences.Source: §6.3 · p. 10

7 Conclusion

SEQ2CAUSE uses one frozen backbone across all four causal-discovery regimes, with the conclusion reporting 91% F1 on sample-level event graphs versus 69% for the strongest baseline. The appendices also discuss limits of multivariate event representations and derive the multivariate Hawkes log-likelihood.

7 Conclusion

  • 91% F1 on sample-level event graphs, versus 69% for the strongest baseline; one frozen backbone populates all four regimes.The conclusion also reports that MDL aggregation bypasses noise when stacking thousands of sequences and recovers population-level boundaries in minutes.Source: §7 · pp. 10–11

A On the Limitation of Multivariate Representation for Event Sequences

A.1 Multivariate Hawkes Process Log-likelihood Derivation

  • Hawkes-process log-likelihood is presented as a sum of per-process log-likelihood terms, then reduced to a total log-likelihood expression.The supplied appendix passages outline the derivation but do not provide the expression’s component steps.Source: §A.1 · pp. 16–17

A.2 Limitations

The limitations discussion covers Hawkes-model scaling, assumptions used in causal discovery, and robustness to selected violations of causal sufficiency. It also states that SEQ2CAUSE does not require stationarity or a parametric transition form beyond the autoregressive model’s expressivity.

A.2 Limitations

  • At |X| = 10^3, Hawkes processes already require 1 million parameters, while likelihood computation can scale as O(|X|^2 · N(T)^2) or O(|X|^2 · N(T)) with exponential decay.The passage also characterizes this regime as computationally prohibitive and statistically prone to overfitting.Source: §A.2 · p. 17

B.1 Definitions

  • Under the stated strong-faithfulness condition, the typical-event example requires causal associations above approximately 0.01 nats, and the noise floors vanish as ϵ →0.Source: §B.1 · p. 18

B.2.1 Causal Sufficiency

  • With 40% of context corrupted, precision remains above 0.8 although recall degrades; under temporal drops, robustness is reported for Pdrop < 0.2.Temporal drops hide intermediate nodes and break conditioning sets; discovery scores decrease as data loss increases.Source: §B.2.1 · p. 18, Figure 3 · p. 18

B.2.2 Temporal Precedence & Instantaneous Effects

  • The method assumes temporal precedence and does not model instantaneous effects; from one trajectory, such effects are not identifiable without additional assumptions, repeated samples, or interventions.Apparent simultaneity is interpreted through the earliest observable temporal ordering, yielding a structure with positive temporal delay.Source: §B.2.2 · pp. 18–19

B.2.3 Consistency Through Time

  • Causal dependencies are assumed invariant across time, although the marginal distribution of events may vary; this assumption is weaker than stationarity.Source: §B.2.3 · p. 19

B.2.4 No Inter-Label Effects

  • Outcome labels are assumed not to cause one another, with their dependence mediated by the event history; the passage says inter-label CI tests are therefore redundant.Source: §B.2.4 · p. 19

B.2.5 Not-required for SEQ2CAUSE

  • SEQ2CAUSE does not require stationarity or a parametric transition form beyond the autoregressive model’s expressivity.The passage notes that autoregressive Transformers can represent non-stationary distributions through contextualized representations.Source: §B.2.5 · p. 19

B.2.6 Broader Impact Limitations

The framework warns that imbalanced event data can leave minority-event conditionals poorly calibrated, causing missed causal edges; Proposition 1 states that its CMI estimator consistently estimates the model-induced CMI as sample size grows.

B.2.6 Broader Impact Limitations

  • Imbalanced logging can poorly calibrate underrepresented event conditionals, leading SEQ2CAUSE to miss edges involving minority event types while recovering frequent-event edges.The resulting graph may appear high-precision while omitting rare-but-critical failure chains.Source: §B.2.6 · p. 20
  • The authors advise treating graphs produced with excess cross-entropy ϵ > 0.1 as unreliable and monitoring that loss, especially for safety-critical deployments.Source: §B.2.6 · p. 20

C.1 Proof of Proposition 1 (CMI Estimator Consistency)

  • As N → ∞, Proposition 1 states that the estimator ˆIN consistently estimates the ϵ-oracle-induced CMI Iθ.Source: §C.1 · p. 20

C.2 Proof of Proposition 2 (Binary Projection of ϵ)

These proofs relate model approximation error to CMI estimation and causal graph recovery, while the comparison passages distinguish Seq2Cause’s conditional-density approach from other causal-discovery paradigms.

C.2 Proof of Proposition 2 (Binary Projection of ϵ)

  • The binary-target approximation error ϵT is bounded by the full categorical error ϵ, with the gap measuring error irrelevant to the target’s binary CI test.Source: §C.2 · pp. 20–21

C.3 Proof of Thm. 1 (CMI Bounded by Model Approximation Error)

  • As sample size grows, the estimator’s stochastic error vanishes, leaving model approximation bias; minimizing excess cross-entropy therefore tightens the upper bound on structural causal error.Source: §C.3 · p. 22

C.4 Proof of Thm. 2 (Soundness of the Sample-Level)

  • Under the stated assumptions, the sample-level CMI inclusion rule recovers parent sets for event and outcome targets, and their union yields the joint sample graph.The proof uses temporal precedence and causal sufficiency; the passage also states recovery for perfect density estimation under standard faithfulness.Source: §C.4 · pp. 22–23

C.5 Proof of Lem. 1 (Data Encoding Cost Deviation under the ϵ-Oracle)

  • When the model cannot certify a positive true probability at an under-resolved position, the log-ratio may be unbounded, so that position is excluded from MDL accumulation.The error-correction quantity ηuv is computable from the frozen model and training diagnostics, with indicators excluding under-resolved positions.Source: §C.5 · pp. 23–24

D.1 Comparison

  • Table 4 defines comparison dimensions including discrete data, nonparametric assumptions, single-stream inference, event-to-event and event-to-outcome targets, scalability, and root-cause analysis.Source: Table 4 · p. 24

D.2 Generative Model for CI-testing

  • Seq2Cause repurposes one pretrained autoregressive model to amortize conditional-density estimation across candidate edges, rather than training a fresh conditional generator for each CI query.The cited generative-CI lineage uses continuous i.i.d. data; Seq2Cause instead targets discrete sequences and extends its analysis to population aggregation with an MDL error-correction term.Source: §D.2 · p. 24

D.3 LLM as Causal Reasoner

  • Unlike approaches that query an LLM as a textual causal oracle, Seq2Cause uses the model’s explicit conditional distribution for CMI tests, with guarantees tied to ϵ rather than prompts.Source: §D.3 · p. 25

D.4 Data Settings

Table 5 contrasts traditional multivariate event-sequence discovery with a single-sequence, session-based setting; Figure 4 illustrates a sample-level graph connecting events to outcomes.

D.4 Data Settings

  • Traditional event-sequence methods use a multivariate structure, while the setting contrasted here uses single sequences in a session-based, NLP-like paradigm.Source: Table 5 · p. 25
  • The table’s central comparison is that standard methods require the multivariate structure and scale poorly in the single-sequence setting.Source: Table 5 · p. 25

E.1 Causal Graph

  • In the sample-level graph example, X1 is the root cause of outcome Y2, and MB1 and MB2 denote the Markov boundaries of Y1 and Y2.Source: Figure 4 · p. 25

F.1 Lagged Effects via Simulated Interventions

Seq2Cause estimates lagged event effects by intervening on intermediate events, then uses parallel CI tests and a sparse-memory variant to address computation. The supplied evaluation passages report vehicle-log causal patterns and comparisons against population-level baselines.

F.1 Lagged Effects via Simulated Interventions

  • For lag ℓ, Seq2Cause estimates an event’s direct effect by intervening on the intermediate events rather than conditioning on them.The approach simulates a controlled direct effect; marginalizing intermediate mechanisms assumes no hidden confounders.Source: §F.1 · pp. 25–26
  • In vehicle diagnostics with about 29,100 event types, Seq2Cause identifies distinct temporal error clusters, including sensor failures at t = 3 preceding mechanical faults at t = 12.The passage also reports a battery issue at t = 17 and describes the analysis as actionable root-cause analysis.Source: §F.1 · p. 26

F.3 Scalability of SEQ2CAUSE

  • Figure 6 depicts parallel CI tests using a broadcasted staircase tensor: adjacent rows provide baseline and intervention distributions for computing CMI.Source: Figure 6 · p. 27
  • Without restricting lag, event-to-event discovery requires a quadratic number of CI tests in sequence length, making sequences longer than 100 difficult even across multiple GPUs.Source: §F.3 · p. 27
  • Bounding lag effects by memory m makes the process an m-order Markov chain and yields linear scaling with sequence length when m ≪ L.Source: §F.3 · p. 27

G.1 Nonlinear SCMs

  • The nonlinear SCM benchmark evaluates summary graphs using intervention-based KL divergence, with an edge included when divergence exceeds τ > 0.05.Source: §G.1 · p. 28

G.2 ϵ in Practice

  • The practical excess-cross-entropy estimate uses held-out validation loss and an approximated entropy floor; the authors report ϵ < 0.05 as sufficient for structural recovery and a phase transition near 0.1.The entropy-floor approximation assumes the validation loss approaches H(P) as capacity and data grow.Source: §G.2 · p. 28

G.3 Population-Level X →X baselines

  • Across observed sequence counts, Seq2Cause outperforms PMI and TF-IDF on the nonlinear SCM benchmark even though those baselines receive oracle F1-maximizing thresholds.Seq2Cause instead uses an MDL-derived threshold without ground-truth access.Source: Figure 7 · p. 29, §G.3 · p. 29

G.4.1 Robustness analysis

SEQ2CAUSE retains strong causal-discovery performance under model error, longer sequences, and delayed dependencies. Population-level results also show benefits from correcting accumulated generative error, while particle and mediator ablations identify sufficient sampling ranges.

G.4.1 Robustness analysis

  • F1 remains >0.80 through delayed effects m=20, one third of the sequence length, and Precision remains high as model fidelity degrades for ϵ < 0.1.Sparse inference uses O(m·L·|X|) memory versus O(L2 · |X|) for the full variant; Precision is ≈0.95 while Recall degrades gracefully on longer sequences.Source: §G.4.1 · p. 30, Figure 9 · p. 31

G.4.3 Sample-Level: Number of Particles N

  • Increasing particles N improves Recall and reduces SHD, but yields no significant changes after N > 256.Figure 11 evaluates sample-level X →X discovery on nonlinear SCMs at |X| = 1000, m = 6, and L = 64.Source: §G.4.3 · p. 30, Figure 11 · p. 33

G.4.4 Population-Level: Number of Counterfactuals M for Mediators

  • Generally, 2 to 4 mediator counterfactuals M are sufficient; the ablation reports no need for larger numbers.Figure 12 evaluates population causal discovery as the number of sampled mediator counterfactuals varies.Source: §G.4.4 · p. 30, Figure 12 · p. 33
  • Precision stays above 0.80 across N ∈[10, 200,000] with correction, while Recall grows; without correction, Precision collapses as sequence evidence accumulates.The reported crossover is at N≈2,000 sequences.Source: Figure 8 · p. 31, NeurIPS Paper Checklist · p. 32

Related papers

Loading 2609.31801v1…