PaperlayerView exact revision on arXiv

Source-linked AI summary

DES-JEPA: Learning a Shared Vocabulary for High-Dimensional Discrete Event Sequences

Hugo Math, Maximilian Forstenhäusler, Christos Anagnostopoulos

arXiv:2610.38445v1cs.AI

TL;DR

Discrete event sequences lack a shared cross-domain vocabulary, and existing methods face limits with high-cardinality event streams. DES-JEPA learns a shared codebook from synthetic processes with unaligned vocabularies, and DES-PFN reuses it for zero-shot next-event prediction. Across eight real-world datasets, the framework is reported to remain competitive at low cardinality and to provide the only reliable non-trivial predictions at hundreds of event types, scaling to 700.

  • Problem

    Discrete event labels lack consistent correspondence across datasets, while real-world streams can have hundreds of event types that challenge current methods.

    Source: §1 · pp. 1–2

  • Method

    DES-JEPA learns a shared codebook of recurring patterns across unaligned processes, which DES-PFN reuses for in-context next-event prediction without fine-tuning.

    Source: §1 · p. 2

  • Results

    Across eight real-world datasets spanning K=3 to K=700, DES-PFN remains competitive at low cardinality and is the only reliable method producing non-trivial predictions at hundreds of event types.

    Source: §4 · p. 5

  • Takeaways & Limitations

    A single frozen codebook supports zero-shot next-event prediction on unseen real-world datasets ranging from three to 700 event types.

    Source: §4 · p. 5

    The codebook size is fixed during pretraining, and the paper evaluates only one-step-ahead prediction, not genuine long-horizon rollouts.

    Source: A Limitations · p. 9

Abstract

from arXiv · show

Temporal foundation models such as Chronos and TimesFM generalize across domains through a shared numerical observation space. In contrast, discrete event sequences (DESs), including operational event logs or patient clinical-event trajectories, have domain-specific event vocabularies whose labels lack a consistent correspondence across domains. Beyond this lack of consistency, the number of distinct events in real-world settings is high enough to challenge both classical temporal point process modeling and current in-context foundation inference models such as FIM-PP. Critically, current foundation models for DESs are limited to a small, fixed number of event types, restricting their applicability to realistic, high-dimensional event streams. To address this, we propose two complementary modules: (i) DES-JEPA, a joint-embedding predictive architecture that learns a shared discrete vocabulary from synthetic stochastic processes with unaligned event vocabularies; and (ii) DES-PFN, a discrete event sequence prior-fitted network that reuses the learned vocabulary for in-context next-event prediction. Without task-specific fine-tuning, this framework achieves competitive zero-shot performance on eight unseen real-world datasets and scales up to 700 distinct event types.

1 Introduction and Related Work

Discrete event sequences have domain-specific labels with no consistent cross-dataset meaning, and current approaches face limits at realistic event cardinalities. The paper proposes a shared codebook learned across unaligned processes and reused for zero-shot next-event prediction.

1 Introduction and Related Work

  • Event labels lack consistent meaning across datasets, while categorical identifiers offer no shared observation space comparable to the numerical values used by temporal foundation models.The introduction identifies vehicle diagnostic logs and electronic health records as examples of discrete event sequences.Source: §1 · p. 1
  • Existing temporal point-process methods can be impractical at hundreds or thousands of event types, while FIM-PP supports at most 22 types.FIM-PP enables in-context learning without fine-tuning, but its event-type limit restricts its direct application to larger vocabularies.Source: §1 · pp. 1–2
  • DES-JEPA learns a shared codebook from synthetic processes with unaligned vocabularies, representing recurring patterns rather than local event symbols.Its approach combines predictive latent learning, vector quantization, and meta-pretraining.Source: §1 · p. 2
  • DES-PFN is evaluated without task-specific fine-tuning on unseen datasets spanning 3 to 700 event types, and is reported as the best-performing model for event sequences.It reuses the shared codebook computed by DES-JEPA.Source: §1 · p. 2

2 Methodology

The methodology learns a shared representation for event sequences whose process-specific alphabets cannot share an embedding table. DES-JEPA builds the codebook, and DES-PFN reuses it to predict events in context and decode predictions to raw event identities.

2 Methodology

  • The problem is to learn a representation invariant to process-specific event types, since each sampled process has its own finite alphabet and no shareable event embedding table.The representation is intended to support downstream modeling of P(x_t|x_<t).Source: §2 · p. 2

2.1 DES-JEPA: A Shared Discrete Codebook

  • DES-JEPA’s student maps history to a representation and predicts codes, while an EMA-updated teacher encodes future windows for nearest-neighbor quantization.The teacher is a shallow prefix of the student encoder; the codebook is Z = {c1, …, cK}.Source: §2.1 · pp. 2–3
  • With w = 2, the student predicts future codes from history; training combines cross-entropy with a β=0.25 commitment term, while inference replaces future tokens to prevent leakage.The teacher receives no gradients, and its commitment term therefore does not back-propagate through it.Source: §2.1 · p. 3
  • Sampling processes during meta-training makes the codebook a finite statistic of process identity; perplexity regularization and usage-EMA revival address codebook underuse and dead codes.The passage characterizes this as amortized Bayesian PFN-style meta-training.Source: §2.1 · p. 3

2.2 DES-PFN: In-Context Autoregressive Next-Event Prediction

  • DES-PFN freezes the codebook, tokenizes queries and contexts, and trains on synthetic processes to autoregressively predict query codes from encoded inputs and prior predictions.The query and context are drawn from the same synthetic DGP; inference updates neither model.Source: §2.2 · p. 3
  • Because quantization is many-to-one, a context-specific kernel estimator decodes predicted codes by similarity-weighting the raw event identities paired with nearby context codes.This permits recovery without an exact predicted-code match in the context; Figure 2 shows the mapping and a single-sequence encode/decode round trip.Source: §2.2 · p. 3, Figure 2 · p. 4

3 Experiments

Across eight real-world datasets, DES-PFN is the only evaluated method with reported results on BGL, MIMIC-IV, and Vehicle, which exceed FIM-PP’s event-type cap. The benchmark uses a shared 128-sequence context for zero-/few-shot methods and reports top-1 accuracy.

3 Experiments

  • The benchmark covers eight real event-type datasets; zero-/few-shot methods receive the same 128 context sequences, and fully trained neural variants are reported as reference upper bounds.Table 1 reports top-1 accuracy averaged over five runs, with 20 queries and sequence length 32; for HDFS, BGL, and MIMIC-IV, its “full” column uses a local split.Source: §3 · p. 4, Table 1 · p. 5
  • DES-PFN is the only method with reported results on BGL, MIMIC-IV, and Vehicle, which FIM-PP cannot evaluate because its architecture is capped at 22 event types.The results span datasets with three to seven hundred event types; some gradient-trained baselines lack usable splits or exceed the training-time budget.Source: §3 · p. 4, Table 1 · p. 5
  • At low cardinality, FIM-PP beats DES-PFN on three of five datasets, while fully trained specialists outperform DES-JEPA on four lower-cardinality easytpp datasets.The passage notes that these specialists require a real held-out test split and per-dataset retraining from scratch to be accurate.Source: §3 · p. 4

4 Conclusion

The conclusion presents DES-JEPA’s shared codebook and DES-PFN’s frozen-codebook inference as a framework for zero-shot prediction across unaligned event vocabularies. It reports coverage from three to 700 event types, while noting that fixed codebook capacity and one-step evaluation leave important questions open.

4 Conclusion

  • DES-JEPA pretrains one shared codebook over synthetic DGPs, giving anchor-free event sequences with no shared vocabulary or numeric scale a common representation.DES-PFN applies that frozen codebook for zero-shot next-event prediction.Source: §4 · p. 5
  • Zero-shot prediction spans eight real-world datasets from K=3 to K=700; the approach remains competitive at low cardinality and the only reliable method producing a non-trivial result at hundreds of event types.Source: §4 · p. 5
  • The codebook capacity is fixed and lacks rate-distortion experiments, while evaluation covers only one-step prediction; long-horizon rollouts remain untested for error accumulation and multiple plausible futures.Source: §4 · p. 5

A Limitations.

The paper identifies fixed codebook size and one-step-only evaluation as limitations, and proposes multi-head prediction of plausible future codes as a next step.

A Limitations.

  • The codebook size is fixed at pretraining rather than adapted to a downstream dataset’s true cardinality.The proposed extension replaces the single-code predictor with a multi-head decoder trained under a winner-takes-all, multiple-choice-learning objective.Source: A Limitations · p. 9
  • DES-PFN has not been evaluated on genuine long-horizon, multi-step rollout.The paper notes that small early errors can compound across steps and a single point forecast poorly represents plausible futures.Source: A Limitations · p. 9

B Architecture Detail

DES-JEPA and DES-PFN were pretrained on one NVIDIA RTX 2080 Ti, while inference measurements used an Apple M2 Pro CPU without GPU acceleration.

B Architecture Detail

  • Pretraining used one NVIDIA RTX 2080 Ti for approximately one day per model; inference was measured on an Apple M2 Pro CPU without GPU acceleration.Source: B Architecture Detail · p. 9

B.1.1 Embeddings

Event representations use statistics of each event type’s own occurrence history rather than raw event-ID lookup embeddings; timestamp and event-count features are both retained.

B.1.1 Embeddings

  • Each event is represented using its own occurrence-history statistics, including surprisal, frequency, seen-before status, and inter-arrival-gap recency and dispersion.Event-count analogues provide step-based timing features; no additional per-event covariates beyond type and time are attached.Source: §B.1.1 · p. 9
  • Highly correlated real-time and step-valued feature pairs are retained because step-valued counterparts remain the only signal when timestamps are missing or unreliable.Examples include causal_freq/local_freq and mean_dt_type/mean_step_gap_type.Source: §B.1.1 · p. 9

B.1.2 Transformer Block

The transformer block uses a causal student, an EMA-updated teacher that encodes future events together, and vector quantization against a codebook.

B.1.2 Transformer Block

  • The student uses a causal Transformer with sliding-window attention, where each token attends only to the 32 preceding tokens.The block also uses RoPE position encoding.Source: §B.1.2 · p. 9
  • The teacher is a bidirectional prefix of two of the student’s six layers, updated only by EMA, and encodes the next two raw events as one unit.The checkpoint uses EMA decay τ=0.999.Source: §B.1.2 · p. 9
  • The teacher’s continuous output is matched to the nearest codebook vector and trained with a combined cross-entropy/VQ-VAE objective.The appendix describes perplexity regularization and dead-code revival to prevent codebook collapse.Source: §B.1.2 · p. 9

B.2.1 Embeddings

DES-PFN reuses the frozen DES-JEPA codebook for event-code embeddings and combines temporal attention with cross-sequence attention. Inter-arrival times are discretized into bins, while optional duplicate-count embeddings mark immediate repeats.

B.2.1 Embeddings

  • DES-PFN uses the frozen DES-JEPA codebook as its event-code embedding table, with a learned linear projection when hidden sizes differ.Source: §B.2.1 · p. 9
  • Inter-arrival times are encoded by binning log(1 + ∆t/scale), and an optional embedding marks immediate event repeats.Source: §B.2.1 · p. 10

B.2.2 Transformer Block

  • DES-PFN alternates causal temporal attention with cross-sequence attention pooled through 32 learned inducing vectors rather than full attention across reference sequences.The causal block uses RoPE and a sliding window of 32.Source: §B.2.2 · p. 10

C.1 Prior-Fitted Network and Synthetic Data Generation

Both stages are pretrained exclusively on synthetic tasks, with fresh processes and active event-type counts spanning 3–1,024. The synthetic suite varies temporal and event-generation structure, and DES-PFN training varies context size and sequence length under a fixed token budget.

C.1 Prior-Fitted Network and Synthetic Data Generation

  • Both stages use only synthetic pretraining data, and every real-world result is zero-shot transfer.Source: §C.1 · p. 10
  • Across tasks, each process draws its own active event-type count; Stage 1 samples one trajectory, while Stage 2 samples reference context and query sequences.Both stages draw active cardinalities log-uniformly from 3 to 1,024; Stage 1 uses length 128 and weights TEMPLATE and HAWKES more heavily.Source: §C.1 · pp. 11–12
  • Other task families include Markov chains of orders 1–5 with optional nonlinear lag interactions, sparse Hawkes processes, and randomly initialized, never-trained LSTMs.Hawkes kernels vary between exponential and power-law forms.Source: §C.1 · p. 11
  • The synthetic suite includes sticky-state HMMs coupling emitted types and Weibull timing, bursty renewal timing with sticky categorical types, and noisy walks over directed event templates.Template edges create near-deterministic event-type coupling, with faster timing along established edges than after restarts.Source: §C.1 · pp. 11–12
  • DES-PFN couples context size and sequence length inversely under a fixed per-batch token budget, varying context count from 16 to 256 and length from 16 to 128.Source: §C.1 · p. 12
  • At inference, the latest observed event receives a code from a repeated-embedding fallback window, unlike training, where positions without genuine future windows are excluded from loss.With W=2, the fallback affects only the most recent position, whose code is therefore systematically less informative than earlier positions.Source: §C.1 · p. 12

D Additional Analysis

The codebook is assessed for broad use across synthetic process families, while the related-work discussion distinguishes shared-vocabulary transfer from other forms of temporal and event-sequence transfer. Its use is not equivalent to uniquely decoding every raw event identifier.

Definition 1 (Codebook Perplexity)

  • Across six DGP families, 601–716 of 768 codes are used per family, while true-code-to-raw-id decode accuracy is 2.5–12.7%.The full training mixture has final perplexity ≈500; high perplexity indicates broad code use, not unique raw-id decoding.Source: Definition 1 (Codebook Perplexity) · pp. 12–13, Figure 4 · p. 13
  • Dead-code revival batches eligible codes for reseeding and uses a usage-share-relative threshold, keeping the criterion scale-invariant as batch size and sequence length grow.The rule is presented as a departure from prior per-code restarts.Source: Definition 1 (Codebook Perplexity) · p. 13

E Related Work

  • Temporal foundation models transfer through shared numerical observations, while PFNs and related models amortize inference without transferring a common discrete vocabulary.Existing DES models learn reusable representations within domains whose event schemas remain stable.Source: E Related Work · p. 13
  • LAMP, HYPRO, and PromptTPP address reasoning-augmented prediction, long-horizon rollout, and continual adaptation within a dataset’s fixed vocabulary, rather than zero-shot transfer across unaligned vocabularies.The paper does not compare against them directly and identifies HYPRO’s long-horizon rollout setting as future work.Source: E Related Work · pp. 13–14
  • DES-JEPA connects vocabulary-symmetry and predictive-discretization directions by learning a shared discrete prediction space across processes with unaligned native alphabets.The related work contrasts this with methods whose alignment is inherited from a common visual or acoustic space.Source: E Related Work · p. 14
Loading 2610.38445v1…