Source-linked AI summary
D-TAIA: Domain-Aware LLM Adaptation for Multi-Task Predictive Process Monitoring
Sjoerd van Straten, Christine Jacob, Marwan Hassani
TL;DR
Existing PPM methods struggle with data scarcity, high process entropy, and distributional shift in joint next-activity and remaining-time prediction. D-TAIA combines parameter-efficient fine-tuning, domain-aware triplet learning, retrieval, and TAIA inference, matching or improving competing baselines across four event logs while leaving direct distribution-shift evaluation open.
Problem
PPM lacks a method that jointly handles data scarcity, high process entropy, and distributional shift for next-activity and remaining-time prediction.
Method
D-TAIA combines LoRA-based FM fine-tuning, DATL pre-training, FAISS retrieval for remaining time, and TAIA inference.
Results
D-TAIA matches or improves upon fine-tuned LLM and RNN baselines across four event logs, with clearest margins at short prefixes and on the highest-entropy log.
Takeaways & Limitations
NLP and computer-vision techniques can support joint PPM using a 10M-parameter backbone, with component contributions varying by dataset entropy.
Takeaways & Limitations
TAIA targets distributional-shift robustness by design, but the paper does not evaluate this directly.
Abstract
from arXiv · showhide
Predictive Process Monitoring (PPM) enables organizations to forecast future process behavior, such as the next activity and remaining time of ongoing cases. In practice, three conditions cause existing methods to degrade, namely data scarcity, high process entropy and distributional shift. While Foundation Models (FMs), especially Large Language Models (LLMs), offer a new paradigm through broad sequential reasoning, adapting them to multi-task PPM under these conditions remains an open challenge. Existing FM-based approaches either lack mechanisms for handling distributional shift or rely on direct regression heads that can be structurally misaligned with continuous time prediction tasks. This paper introduces D-TAIA (Domain-aware Training and Attention-based Inference Architecture), a framework for a joint next activity and remaining time prediction task via parameter-efficient fine-tuning of an FM backbone. Our approach combines domain-aware triplet loss (DATL) pre-training with FAISS-based nearest neighbor retrieval for remaining time prediction, and adopts the TAIA inference strategy to preserve pre-trained sequential reasoning during fine-tuning. Evaluated across four real-world event logs, D-TAIA consistently shows SOTA or competitive performance compared to a fine-tuned LLM and a recurrent neural network baseline. Ablation studies confirm that techniques from NLP and computer vision can be transferred effectively to PPM with only a 10M-parameter backbone, though component contributions vary by dataset entropy.
1 Introduction
PPM methods degrade under data scarcity, high process entropy, and distributional shift, while existing FM-based approaches do not jointly address these conditions. D-TAIA combines parameter-efficient FM fine-tuning, domain-aware retrieval, and TAIA inference for joint next-activity and remaining-time prediction.
- Challenges: Data scarcity, high process entropy, and distributional shift are three conditions that degrade existing PPM methods.They respectively limit process representation learning, increase continuation uncertainty, and violate the i.i.d. assumption.
- Research Gap: Existing methods address individual conditions, but no approach handles all three simultaneously.
- Proposed Framework: D-TAIA uses LoRA fine-tuning with DATL pre-training and FAISS retrieval to improve joint next-activity and remaining-time prediction.The design targets data scarcity and high process entropy.
- Proposed Framework: TAIA inference preserves pre-trained sequential reasoning during fine-tuning on small logs to address distributional shift architecturally.
- Evaluation: Across four real-world event logs, D-TAIA matches or improves upon fine-tuned LLM and RNN baselines, especially at short prefixes and on the highest-entropy log.
2 Preliminaries and Problem Definition
The paper formulates joint next-activity and remaining-time prediction from observable trace prefixes. Its difficulty arises from insufficient data, continuation uncertainty, and differences between training and test distributions.
- Process Representation: An event log is a finite collection of traces, each trace being a timestamped sequence of activity-labeled events.
- Prediction Tasks: Given an observed prefix, the next-activity task predicts the activity immediately following that prefix.
- Prediction Tasks: Remaining-time prediction estimates the interval from the final observed event to the completed trace’s final event.It is defined as RT(σ, k) = tL − tk.
- Objective: The central objective is joint prediction of next activity and remaining time from a single model.
- Challenges: Data scarcity limits process representation learning, especially with rare activities and cross-validation partitioning effects.
- Challenges: High process entropy measures the spread of plausible next-activity continuations, while distributional shift occurs when test and training conditional distributions differ.
3 Related Work
Prior PPM methods generally target subsets of the data, entropy, generalization, or task dimensions. The identified gap is a joint framework addressing all three conditions for next-activity and remaining-time prediction.
- PPM Approaches: Non-FM methods typically address one condition and often one prediction task at a time.
- PPM Approaches: Transfer learning addresses data scarcity but requires a structurally related source log and focuses on outcome prediction.
- PPM Approaches: Entropy-aware attention and graph-based control-flow modeling target high-entropy prediction, but graph modeling can degrade on sparse logs.
- FM-Based Approaches: Prompt-based FM approaches support data-scarce generalization but do not target the joint next-activity and remaining-time tasks.
- FM-Based Approaches: Parameter-efficient LLM fine-tuning supports joint prediction, but prior work uses a point-estimate time head without mechanisms for high entropy or distributional shift.
- Research Gap: The research gap is a method simultaneously addressing data scarcity, high process entropy, OOD generalization, and joint prediction.
4 Method
D-TAIA is a three-stage architecture that learns domain-aware prefix embeddings, fine-tunes an LLM with LoRA for two tasks, and combines direct and retrieval-based time estimates. TAIA inference removes FFN updates before evaluation to preserve pretrained behavior under shifted prefixes.
- Framework Overview: D-TAIA processes prefixes through pre-training, supervised fine-tuning, and inference stages.
- Data Preparation: Each prefix is paired with its next activity and remaining time, padded or truncated to 20 events with 19 numerical features per event.
- Stage 1: Pre-Training: DATL trains prefix embeddings so samples cluster by remaining-time behavior rather than domain, then stores normalized embeddings and times in a frozen FAISS index.
- Stage 2: Supervised Fine-Tuning: LoRA fine-tunes attention and FFN sublayers while a shared hidden state feeds separate activity-classification and time-prediction heads.
- Stage 3: Inference: Before evaluation, all FFN LoRA adapters are set to zero so attention-based sequential reasoning remains while fine-tuned FFN memory is discarded.
- Stage 3: Inference: The FusionGate combines direct and retrieval-based time estimates as rtfinal = β rtdirect + (1−β) rtretrieved, with β fixed rather than learned.
5 Experimental Setup
The evaluation uses four public event logs, three performance metrics, two baselines, and fixed temporal data splits with repeated-seed reporting.
- Datasets and Metrics: Four publicly available BPI Challenge 3 event logs span varying data availability and process complexity.
- Datasets and Metrics: Macro-F1, RT MAE in days, and wall-clock running time measure activity prediction, remaining-time error, and computational cost.
- Baselines: D-TAIA is compared with FT-LLM and MT-RNN for joint next-activity and remaining-time prediction.
- Baselines: FT-LLM uses the same backbone as D-TAIA when backbone capacity is varied, isolating the effects of D-TAIA’s added components.
- Implementation Details: Experiments use a fixed temporal 65/15/20 train-validation-test split, five seeds, and 95% confidence intervals across runs.
6 Evaluation Results
Across four event logs, D-TAIA generally outperforms FT-LLM and MT-RNN, with the clearest benefits on high-entropy data, short prefixes, and remaining-time prediction.
- D-TAIA achieves higher Macro-F1 and lower MAE than FT-LLM across all backbones and datasets.
- Backbone Sensitivity: Scaling the backbone improves both metrics but increases runtime, with the largest Macro-F1 gain on high-entropy BPI2015_2.Macro-F1 rises 13.7% from Tiny-LLM to Llama3.2 on BPI2015_2, compared with 3–6% on the other logs.
- Training Data Sensitivity: D-TAIA has the best point estimate for Macro-F1 and RT MAE at every training-data budget across all four logs.At 20% training data, overlapping confidence intervals reflect greater estimation uncertainty at small sample sizes.
- Prefix Length Sensitivity: D-TAIA’s advantage over FT-LLM is largest at short prefixes and narrows toward longer prefixes, while remaining proportionally largest on high-entropy BPI2015_2.The Macro-F1 gap at bucket 1 is 3.0–6.0 percentage points, and RT MAE follows the same pattern.
- Ablation Study: Removing DATL causes the largest point-estimate drop, whereas removing FAISS affects MAE more than Macro-F1 and removing TAIA has the smallest effect.DATL removal is the only variant significantly distinguishable from the full model on both metrics in the high-entropy setting.
7 Conclusion and Future Work
D-TAIA addresses joint next-activity and remaining-time prediction under data scarcity and high process entropy using retrieval, DATL pre-training, and attention-based inference. It matches or improves upon fine-tuned LLM and RNN baselines across four logs, while robustness to distributional shift remains unevaluated and several design choices are left for future work.
- D-TAIA combines FAISS retrieval, DATL pre-training, and attention-based inference for joint prediction under data scarcity and high process entropy.
- Across four event logs, D-TAIA matches or improves upon a fine-tuned LLM and an RNN baseline, with clearest margins at short prefixes and on the highest-entropy log.
- Robustness to distributional shift is a design target, but the paper does not evaluate it directly and treats it as open.
- Future Work: Future work includes testing TAIA on topology-aware backbones, learning the FusionGate weight from entropy and retrieval confidence, and injecting retrieved prefixes into the context.