PaperlayerView exact revision on arXiv

Source-linked AI summary

Foundation Models for Event Prediction Across Heterogeneous Event Logs

Fabian Spaeh, Sean Jaffe, Shandian Zhe, Bin Shen

arXiv:2610.17503v1cs.LGcs.AI

TL;DR

Heterogeneous event logs make it difficult to build a predictor that transfers directly to unseen systems. FEP uses in-context event-level inference with representations designed to avoid persistent event-ID semantics. Across held-out systems, it often approaches strong local models while substantially outperforming foundation-model baselines.

  • Problem

    Event IDs can refer to unrelated activities across heterogeneous systems, complicating generalization from one event log to another.

    Source: §3.4 · p. 6

  • Method

    FEP aggregates observed history-to-next-event examples with learned cross-attention and uses episode-varying event representations and time normalization.

    Source: §1 · p. 2, §3.2 · p. 5

  • Results

    FEP often approaches strong local models without parameter adaptation and substantially outperforms competing foundation models.

    Source: §1 · p. 2, §6 · p. 11

  • Takeaways & Limitations

    The results support using a single pretrained FEP model to predict next events on unseen heterogeneous logs without target-specific parameter adaptation.

    Source: §6 · p. 11

    FEP models event types and timestamps but does not incorporate additional event attributes or sequence-level metadata.

    Source: §5 · p. 11

Abstract

from arXiv · show

Event logs are ubiquitous across real-world domains, including business processes, manufacturing workflows, healthcare, and many other operational systems. Yet event logs can differ substantially in their event semantics, label spaces, interaction patterns, and temporal scales, making it difficult to build a single foundation model that generalizes directly to previously unseen systems. We address this challenge with the Foundation Event Predictor (FEP), a simple and scalable in-context architecture that performs next-event prediction through learned, event-level nonparametric inference. Causal self-attention encodes context and target histories into key and query representations, respectively, while the observed next context events are encoded as values. Cross-attention then performs learned, relevance-weighted aggregation of next-event outcomes over all context transitions, allowing the entire nonparametric prediction procedure to be expressed as standard attention operations and efficiently parallelized with modern kernels such as FlashAttention. To promote cross-system generalization, we introduce permutation-robust embedding learning and random fixed embeddings to decouple event IDs from persistent semantics, together with empirical quantile transforms and robust normalization to handle diverse temporal scales. We pretrain FEP on 696 real-world event datasets spanning heterogeneous event systems and evaluate it on 50 held-out datasets unseen during pretraining. Across these datasets, FEP generalizes effectively to new event systems without parameter adaptation, frequently approaching the performance of strong models trained separately on each target dataset while substantially outperforming existing foundation-model baselines, particularly on challenging prediction tasks.

1 Introduction

FEP targets next-event prediction across heterogeneous event logs, where differing event meanings, vocabularies, and temporal scales make direct generalization difficult. It learns from event-level context transitions and uses episode-varying event representations and time normalization to generalize across systems.

1 Introduction

  • Event logs span many operational domains, and predicting the next event and its timing supports understanding, monitoring, and anticipating process evolution.Applications include process monitoring, optimization, resource planning, anomaly detection, and proactive intervention.Source: §1 · pp. 1–2
  • FEP treats observed history-to-next-event transitions as predictive examples, using causal self-attention for context keys and target queries and cross-attention to aggregate next-event outcomes.The output is decoded into the next event type and inter-arrival time.Source: §1 · p. 2
  • Permutation training and random fixed embeddings discourage persistent event-ID semantics, while empirical quantile transforms and robust log-based normalization address heterogeneous time scales.Both embedding strategies encourage inference from observed context rather than dataset-specific label identities.Source: §1 · p. 2
  • Across difficulty levels, FEP approaches the strongest local models and substantially outperforms competing foundation models, while also outperforming an architecture-matched local counterpart on average.Evaluation spans 696 pretraining datasets and 50 held-out datasets unseen during pretraining.Source: §1 · p. 2

2 Related Work

Prior event-sequence models commonly fit a separate predictor to each system, while newer foundation approaches use in-context examples for unseen logs. FEP instead learns to aggregate event-level history-to-next-event pairs through cross-attention for direct next-event prediction.

2 Related Work

  • Traditional temporal point-process models capture event dependencies or next-time distributions, but typically train separately within a fixed event-type vocabulary.Examples include Hawkes processes, RMTPP, Neural Hawkes Processes (Mei and Eisner, 2017), and Transformer-based models (Zuo et al., 2020; Yang et al., 2022).Source: §2 · p. 3
  • Predictive process monitoring models forecast future behavior from running-case prefixes, but generally train a separate model for each event log.They address next-activity, next-event-time, and remaining-time prediction (Evermann et al., 2017; Tax et al., 2017; Bukhsh et al., 2021).Source: §2 · p. 3
  • FEP differs from temporal point-process approaches by training on heterogeneous real-world logs and aggregating observed history-to-next-event examples for direct prediction rather than estimating an intensity function.FIM-PP instead infers Hawkes-like conditional intensities under a synthetic Hawkes-family prior (Berghaus et al., 2026).Source: §2 · p. 3
  • In-context predictive process-monitoring models use prototype classification for next activity and kernel regression for remaining case time; FM-v2 makes retrieval central to training and inference.Remaining case time is time until process completion, not time to the next activity.Source: §2 · p. 3
  • FEP unifies learned relevance weighting and outcome representation over event-level examples, with standard attention enabling parallel computation using optimized kernels.The related approaches also use in-context examples and locally defined activity vocabularies; FEP predicts both event type and inter-arrival time (Dao et al., 2022; Dao, 2024).Source: §2 · pp. 3–4

3 Methodology

FEP constructs event-level examples from context histories and uses cross-attention to aggregate observed next-event outcomes for target histories. Context-based time normalization and episodic event-ID randomization address differences in temporal scales and event vocabularies.

3.1 Problem Setting

  • FEP targets one model for unseen systems with varied event semantics, label spaces, interaction patterns, and temporal scales, using context sequences to predict target next events without adaptation.Source: §3.1 · p. 4

3.2 Event-Level In-Context Prediction

  • FEP decomposes context data into event-level history-to-next-event examples, using shared causal Transformer encodings to represent context histories and target histories as keys and queries.Source: §3.2 · p. 4
  • Cross-attention learns relevance-weighted aggregation over context transitions, exploiting fine-grained supervision through standard attention operations that can use FlashAttention.Source: §3.2 · pp. 5–6

3.3 Robust Representation of Time

  • FEP normalizes inter-arrival times using context-only statistics, with empirical quantile transforms or robust log-time normalization that resists extreme waiting times.Source: §3.3 · p. 6

3.4 Removing Spurious Semantics from Event IDs

  • To avoid persistent event-ID semantics across heterogeneous logs, permutation training remaps learned embeddings each episode, while random fixed embeddings generate fresh representations.Source: §3.4 · p. 6

3.5 Training Across Event Systems

  • Training samples context and query sequences across systems, accumulates gradients across systems, and updates parameters after each episode while releasing each query batch's computation graph.Source: §3.5 · p. 6

4 Experiments

Across 50 held-out datasets, FEP performs strongly against foundation-model baselines and remains competitive with the best locally trained models. Ablations show gains from larger contexts and broader pretraining, while model size has a moderate optimum.

4.1 Generalization Performance

  • FEP is pretrained on 696 real-world datasets and evaluated on 50 held-out datasets without overlap in dataset or process identity.Comparisons include separately trained local methods and existing event-data foundation models.Source: §4.1 · p. 7
  • Evaluation reports next-event-type accuracy and inter-arrival-time NMAE, with difficulty groups used for stratified analysis while all 50 datasets remain in the overall evaluation.The test set uses the newest 15% of traces ordered by sequence start time.Source: §4.1 · p. 7
  • Across 50 test datasets, FEP has the highest Bradley–Terry scores among foundation models and remains competitive with local baselines.The table reports overall pairwise ratings, with higher scores indicating better performance.Source: Table 2 · p. 9, §4.1 · p. 9

4.2 Ablation Studies

  • Increasing context size typically improves accuracy and NMAE, with diminishing returns and no evident attention dilution over the tested range.Even a moderate number of traces provides many event-level examples for cross-attention.Source: §4.2 · p. 9
  • Pretraining on 30 datasets substantially improves both metrics over 3 or 10, gains are modest from 30 to 50, and 696 yields further improvement.Time prediction is unstable with 10 pretraining datasets.Source: §4.2 · pp. 9–10

4.1 Generalization Performance

  • FEP generally approaches the best locally trained model without target-specific adaptation, although a small number of datasets show larger gaps.Figure 2 compares FEP with the best local model separately for each test dataset.Source: §4.1 · p. 9, Figure 2 · p. 10

4.2 Ablation Studies

  • FEP performs well with shallow networks; moderate widths around dmodel = 64–128 largely saturate performance, while excessive depth or width 256 degrades it.Increasing depth from one to a few layers provides modest gains.Source: §4.2 · p. 10

5 Limitations

FEP currently predicts using event types and timestamps, leaving additional event attributes and sequence-level metadata outside its scope.

5 Limitations

  • FEP models event types and timestamps but omits additional event attributes and sequence-level metadata.These auxiliary features can provide predictive signals, but their schemas, ranges, cardinalities, availability, and noise vary across logs.Source: §5 · p. 11
  • Designing flexible representations for heterogeneous attributes while preserving cross-system generalization remains an open problem.The paper leaves this extension to future work.Source: §5 · p. 11
  • Additional event attributes and sequence-level metadata are not incorporated in the current framework.Source: §5 · p. 11

6 Conclusion

The conclusion presents FEP as an in-context model for unseen heterogeneous event logs, with encouraging next-event prediction without parameter adaptation. It often approaches target-trained local models, while future work includes adding features, synthetic pretraining data, and uncertainty calibration.

6 Conclusion

  • FEP predicts next events on unseen heterogeneous logs without parameter adaptation, with encouraging performance.Source: §6 · p. 11
  • FEP often approaches the performance of strong models trained specifically on target datasets.Source: §6 · p. 11
  • Future work will incorporate heterogeneous event features, expand pretraining with synthetic data, and investigate uncertainty calibration.Source: §6 · p. 11

A Experimental Details

The experimental details specify FEP’s episodic training and implementation, the separately trained local baselines, and the protocols used to evaluate models on held-out datasets.

A Experimental Details

  • FEP uses hidden dimension 128, four attention heads and layers, feed-forward dimension 512, and dropout 0.05.The maximum per-system training pool is 2,560 traces; larger systems are resampled each episode.Source: A Experimental Details · p. 15
  • FEP is trained for 500 episodes with AdamW, learning rate 10^-3, and weight decay 10^-5; distributed workers average gradients and use FlashAttention-2.Source: A Experimental Details · p. 15
  • Local Transformer, GRU, Neural Hawkes, Transformer Hawkes, and IFTPP baselines are trained separately on each target dataset.Validation accuracy and time-MAE ranks guide epoch and configuration selection; final metrics use five independent runs.Source: A Experimental Details · p. 15
  • FEP and frozen foundation baselines use up to 2,048 target traces, while fine-tuned FIM-PP uses at most 500 traces.When datasets exceed 22 activity classes, FIM-PP maps the 21 most frequent activities and all remaining ones to a single “other” class.Source: A Experimental Details · p. 16
  • Each FEP episode samples context and query sequences, encodes context histories as keys and next-event values, then cross-attends query representations for predictions.Gradients accumulate across episodes before model parameters are updated.Source: A Experimental Details · p. 16

B.1 Inference Efficiency and Scalability

The section benchmarks cached-context inference with standard attention and FlashAttention-2. FlashAttention-2’s advantage grows at larger context sizes, where it reduces latency and memory use while increasing throughput.

B.1 Inference Efficiency and Scalability

  • The benchmark compares standard attention, which materializes attention matrices, with FlashAttention-2’s exact, memory-efficient attention implementation.Source: §B.1 · p. 17
  • Measurements use length-100 query traces yielding 99 predictions, with context encoding excluded because cached representations are reused.At fixed query length, exact cross-attention computation remains linear in context size, while FlashAttention-2 avoids materializing full attention matrices.Source: §B.1 · pp. 17–18
  • At context sizes through 512 sequences, standard attention is approximately 3–4% faster, while FlashAttention-2 already saves memory.Source: §B.1 · pp. 17–18
  • At 16,384 context sequences, FlashAttention-2 reduces latency from 17.616 to 10.733 ms, increases throughput from 56.8 to 93.2 queries/s, and cuts peak memory from 7.420 to 4.193 GiB.Source: §B.1 · p. 18

B.2 Ablations on Event-ID Representations

The event-ID ablation favors approaches that prevent numerical IDs from acquiring persistent semantics. Across the full-data configurations, performance is close across the proposed representation and time-normalization choices.

B.2 Ablations on Event-ID Representations

  • The ablation compares random fixed embeddings, learned embeddings with per-episode event-ID permutation, and learned embeddings without permutation.Variants use 33 pretraining datasets, 23 held-out datasets, and robust log normalization for inter-arrival times.Source: §B.2 · p. 18
  • Learned embeddings without permutation reduce accuracy by nearly four percentage points and increase NMAE, while random fixed and permutation-trained embeddings perform strongly and similarly.Permutation training performs slightly better overall.Source: §B.2 · p. 18

B.3 Combinations of Event-ID Representation and Time Normalization

  • With full-data pretraining, all four event-ID and time-normalization combinations perform closely across difficulty levels, with no clear winner.The authors report robustness to the choice between the two proposed event-ID representations and time-normalization strategies.Source: §B.3 · p. 18, Table 7 · p. 19

B.4 Dataset-wise Effect of Cross-Dataset Pretraining

Across held-out datasets, FEP generally improves event-type accuracy and matches or lowers time-prediction error compared with FEP-Local. The gains are broadly distributed, supporting the benefit of cross-dataset pretraining for generalization to unseen event systems.

  • Most dataset-wise comparisons favor FEP over FEP-Local in event-type accuracy and show lower or comparable NMAE.The prose reports substantial accuracy improvements on many datasets.Source: §B.4 · p. 19
  • FEP’s gains over FEP-Local are broadly distributed across test datasets rather than driven by only a few datasets.Source: §B.4 · p. 19
  • FEP substantially outperforms FEP-Local on average across all three difficulty levels, demonstrating a benefit of cross-dataset pretraining for unseen event systems.Source: §B.4 · p. 19

C Dataset Information

The dataset collection spans multiple domains, with environmental monitoring held out from pretraining. Its test sets also differ from pretraining data in temporal and activity-count distributions, particularly for hard datasets.

  • Test datasets are checked for overlap with pretraining data by dataset and process identity, including duplicate or derived logs, and their training splits are capped at 10,000 traces.Source: C Dataset Information · p. 19
  • Datasets from public online repositories are processed individually into event-sequence representations; when a dataset has more than 100 activity types, the 99 most frequent are retained and the remaining types merged into a general “other” activity.Source: C Dataset Information · p. 19
  • Test datasets show distribution shifts from pretraining, especially in inter-arrival times and activity counts for the hard group.The distributions compared include inter-arrival times, trace durations, activity counts, and trace lengths.Source: C Dataset Information · pp. 19, 21, Figure 5 · p. 25
  • Environmental monitoring is absent from pretraining, and the collection includes 696 pretraining datasets and a catalog of 50 real-world test datasets.The datasets are grouped by domain in Table 9; Table 10 describes the test datasets.Source: Table 9 · p. 21, C Dataset Information · p. 21
  • Table 9 groups real-world pretraining and evaluation datasets by domain, while Table 10 catalogs 50 real-world test datasets with descriptions and applicable preprocessing details.For Table 10, trace length and duration (in days) are averaged over all traces within each dataset.Source: Table 9 · p. 21, C Dataset Information · p. 21, Table 10 · p. 21

Related papers

Loading 2610.17503v1…