PaperlayerView exact revision on arXiv

Source-linked AI summary

Outcome-Oriented Predictive Process Monitoring: Review and Benchmark

Irene Teinemaa, Marlon Dumas, Marcello La Rosa, Fabrizio Maria Maggi

arXiv:1707.06766v4cs.AI

TL;DR

Outcome-oriented predictive process-monitoring methods were difficult to compare because studies used different datasets, settings, measures, and baselines. The paper reviews and categorizes these methods, then benchmarks eleven techniques across 24 tasks and nine event logs. The benchmark finds that lossy aggregation encoding is most reliable and accurate by AUC, while concept drift significantly affects all tested methods in two datasets.

  • Problem

    Different studies evaluated outcome-oriented predictive-monitoring methods with inconsistent datasets, experimental settings, evaluation measures, and baselines, limiting comparability.

  • Method

    The paper conducts a systematic review, develops a taxonomy based on trace bucketing and sequence encoding, and evaluates identified methods in a unified framework.

  • Results

    Lossy aggregation encoding produced the most reliable and accurate AUC results, while concept drift in data attributes significantly affected all tested methods in two datasets.

  • Takeaways & Limitations

    Researchers and practitioners should consider aggregation-based encoding and check for concept drift when applying predictive-monitoring methods.

  • Takeaways & Limitations

    The evaluation tested only one clustering method, one state abstraction, and four classification algorithms, so other configurations may outperform the tested settings.

Abstract

from arXiv · show

Predictive business process monitoring refers to the act of making predictions about the future state of ongoing cases of a business process, based on their incomplete execution traces and logs of historical (completed) traces. Motivated by the increasingly pervasive availability of fine-grained event data about business process executions, the problem of predictive process monitoring has received substantial attention in the past years. In particular, a considerable number of methods have been put forward to address the problem of outcome-oriented predictive process monitoring, which refers to classifying each ongoing case of a process according to a given set of possible categorical outcomes - e.g., Will the customer complain or not? Will an order be delivered, canceled or withdrawn? Unfortunately, different authors have used different datasets, experimental settings, evaluation measures and baselines to assess their proposals, resulting in poor comparability and an unclear picture of the relative merits and applicability of different methods. To address this gap, this article presents a systematic review and taxonomy of outcome-oriented predictive process monitoring methods, and a comparative experimental evaluation of eleven representative methods using a benchmark covering 24 predictive process monitoring tasks based on nine real-life event logs.

1 INTRODUCTION

Predictive process monitoring forecasts the future states or outcomes of ongoing business-process cases from incomplete traces and historical logs. This paper addresses poor comparability among outcome-oriented methods through a systematic review, taxonomy, and unified benchmark.

  • Introduction: Business processes comprise inter-related events, activities, and decisions involving actors and objects that lead to a customer-valued outcome.An execution is a case; in order-to-cash, each purchase order creates a case.
  • Introduction: Predictive monitoring extends traditional dashboards by forecasting remaining time, next activities, or categorical case outcomes for ongoing process executions.Possible outcomes include satisfactory closure, cancellation, withdrawal, or timely versus late delivery.
  • Introduction: Outcome-oriented predictive process monitoring classifies ongoing cases by their possible business outcomes, but existing studies use inconsistent datasets, settings, measures, and baselines.These differences leave methods difficult to compare directly.
  • Introduction: The paper conducts a systematic literature review, proposes a taxonomy, and evaluates eleven representative methods on 24 tasks from nine real-life event logs.The benchmark is intended to provide a unified basis for comparison.
  • Introduction: The benchmark is released as an open-source framework that supports running the full evaluation and adding methods or datasets.Researchers can configure and extend the framework with minimal effort.

2 BACKGROUND

The paper models event logs as traces of case-specific event records and frames outcome prediction as early sequence classification. Classifiers operate on feature vectors encoded from incomplete trace prefixes and their control-flow or data attributes.

  • Event logs and traces: An event record contains an activity name, case identifier, timestamp, and optional event or case attributes with their values.Event and case attributes may be numeric, categorical, or textual and require different preprocessing.
  • Event logs and traces: A trace is a non-empty sequence of events whose events all refer to the same case.The paper illustrates traces using patient-treatment cases with activity, timing, and attribute values.
  • Prefixes and outcomes: An event log is a set of completed traces, while predictive monitoring uses incomplete traces from ongoing cases.A prefix contains the first l events of a completed trace.
  • Prefixes and outcomes: Outcome-oriented monitoring predicts a trace’s categorical class label from completed labeled cases.The labeling function maps traces to outcomes in a finite label domain, such as Y = {0, 1}.
  • Encoding and classification: A sequence encoder transforms a partial trace into a fixed-dimensional feature vector that a classifier uses to estimate its class label.Features can represent activity order and occurrence or event and case attributes.
  • Encoding and classification: Training uses prefixes collected in a prefix log, with length-filtered logs restricting prefixes to a maximum size.The broader machine-learning framing is early sequence classification: predicting the eventual label from a sequence prefix.

3 SEARCH METHODOLOGY

The study uses a systematic literature review to identify and filter outcome-oriented predictive-monitoring research, then separates primary from subsumed studies for taxonomy and benchmarking. The process yielded 14 relevant studies, including seven primary and seven subsumed studies.

  • Review procedure: The review began with research questions, database search strings, inclusion and exclusion criteria, and classification of relevant studies as primary or subsumed.The procedure followed an established systematic-literature-review approach.
  • Research questions: The survey focuses on how to train models that predict incomplete traces’ outcomes and asks which methods exist, how to categorize them, and how they perform comparatively.Outcome-oriented monitoring was selected as a bounded scope within the broader predictive-monitoring literature.
  • Search and selection: The search used three predictive-process-monitoring phrases in Google Scholar and retrieved 93 non-duplicate papers in August 2017.The phrases covered predictive process monitoring, predictive business process monitoring, and business process prediction.
  • Search and selection: Two authors independently assessed studies, compared their decisions, and used a third author to mediate inconsistencies.Inclusion criteria were applied before deeper exclusion-criteria assessment.
  • Search and selection: The review retained studies concerning business-process prediction, outcome-oriented classification, and event-log inputs containing essential process information.Exclusions covered non-predictive monitoring, non-outcome targets, tailored labeling functions, and methods lacking required event-log elements.
  • Primary and subsumed studies: 14 relevant studies remained after exclusion screening, and seven primary plus seven subsumed studies were selected for the benchmark-oriented analysis.Subsumption covered newer or more extensive versions, insufficient methodological improvement, or case-study and tool contributions superseded by later method descriptions.

4 ANALYSIS AND TAXONOMY

The taxonomy organizes predictive process monitoring around an offline training phase and an online prediction phase, covering prefix filtering, bucketing, encoding, and classification. It also identifies alternative bucketing and encoding strategies, while noting an exception that trains models online.

  • Offline phase: The offline workflow extracts and filters case prefixes, assigns them to buckets, encodes features, and trains a classifier for each bucket.Prefixes may be filtered by length or gaps before bucket-specific feature encoding and classifier training.
  • Online phase: The online workflow determines the appropriate bucket for a running trace, encodes its features, and applies the corresponding classifier to produce a prediction.The online phase reuses the buckets and classifiers constructed offline.
  • Prefix selection and filtering: Prefix filtering can use a maximum length or periodic gaps, and this choice is applicable across methods rather than inherent to any particular method.Gap filtering retains prefixes at lengths such as 1, 6, 11, 16, and 21 for a gap of 5.
  • Trace bucketing: Bucketing ranges from a single global bucket to state-based, clustering-based, and nearest-neighbor approaches with different classifier-training arrangements.The KNN approach selects similar historical prefixes and trains a classifier at run time for each running case.
  • Trace bucketing: State-based approaches can map prefixes to process states, while a directly-follows graph maps each prefix to the state represented by its last activity.The surveyed DFG representation omits transition probabilities because predictions are intended for running cases regardless of frequency.
  • Sequence encoding: Sequence encoding converts variable-length prefixes into fixed-length feature vectors through trace abstraction and feature extraction, balancing generality against information loss.Case attributes can be added directly, with categorical values represented using one-hot encoding.

Summary.

The taxonomy organizes outcome-oriented predictive process monitoring methods around trace bucketing and sequence encoding, while treating classifiers and prefix filtering as orthogonal choices. These design choices still affect predictive performance and are evaluated modularly.

  • Classification algorithms: Existing methods commonly use decision trees for interpretability, random forests for higher accuracy, and gradient-boosted trees that often outperform random forests.
  • Orthogonal choices: Prefix filtering and classification algorithms are orthogonal to the taxonomy, yet their specific choices remain important for prediction quality in particular settings.
  • Experimental considerations: Fair comparisons should provide every method with the same, preferably largest, set of case and event attributes because combining control flow and data payload can increase predictive power.
  • Taxonomy dimensions: The taxonomy compares methods by how prefix traces are divided into buckets and how event attributes are transformed into features.
  • Taxonomy dimensions: The modular taxonomy contains 11 approaches derived from seven primary studies, including combinations not previously used in the literature.
  • Taxonomy scope: Double-bucketing methods are excluded from the taxonomy because many small buckets can leave classifiers with too few training instances to learn meaningful patterns.

5 BENCHMARK

The benchmark evaluates 11 predictive process monitoring approaches using accuracy, earliness, and computation time. It is implemented as an open-source, tunable, and extensible Python framework.

  • Benchmark design: 11 approaches are benchmarked using prediction accuracy, earliness, and computation time to compare their relative performance.
  • Implementation: The experiments use an open-source, tunable, and extensible Python framework built with Python 3.6 and scikit-learn.

5.1 Datasets

The benchmark comprises 24 outcome-prediction tasks derived from nine real-life event logs, selected to include both static case attributes and dynamic event attributes. The datasets vary substantially in size, case length, and class balance.

  • Dataset selection: The benchmark includes eight publicly available logs and one private insurance log, while excluding logs without both static case and dynamic event attributes.
  • Dataset construction: 24 prediction tasks are formulated from nine original event logs, with separate tasks created when logs use multiple outcome-labeling functions.
  • Labeling functions: BPIC2011 contributes four LTL-based tasks, whose labels indicate whether each trace satisfies or violates a specified temporal rule.
  • Labeling functions: Some LTL labels become irreversible before completion, so BPIC2011 task cases are cut before decisive events; other tasks prohibit trace cutting because labels remain unresolved until completion.
  • Event logs: The datasets cover manufacturing, insurance claims, sepsis care, loan applications, hospital billing, and traffic fines, with task-specific binary outcome definitions.
  • Dataset characteristics: The smallest log contains 220 cases and the largest 129615, while BPIC2011 has cases up to 1814 events and traffic cases range from 2 to 20 events.
  • Feature preparation: Timestamp preprocessing derives month, weekday, hour, event durations, case duration, and event position as predictive features.

5.2 Experimental set-up

The evaluation compares predictive monitoring methods by accuracy, earliness, and execution time under a temporal train-test setting. It uses AUC, F-score, prefix-length analyses, four classifiers, and optimized bucketing, encoding, and model hyperparameters.

  • Evaluation objectives: Accuracy and earliness are the primary prediction-quality dimensions, because useful predictions should be both reliable and available early enough to support action.The study formulates RQ3.1 around comparing methods on both dimensions.
  • Evaluation measures: AUC evaluates ranking quality and remains threshold-independent and unbiased under highly imbalanced class labels.AUC expresses the probability that a positive case receives a higher score than a negative case.
  • Evaluation measures: Earliness is assessed either across exact prefix lengths or by identifying how quickly a method reaches an acceptable accuracy level.Shorter prefixes at the acceptable accuracy level indicate earlier useful prediction.
  • Evaluation objectives: Execution-time analysis separates offline model construction from online prediction and excludes hyperparameter-selection time from offline measurements.Offline time includes prefix-log construction, bucketing, encoding, and classifier training.
  • Experimental protocol: The experiments use a temporal split, with the first 80% of cases for parameter selection and training and the remaining 20% for evaluation.Cases are ordered by start time and split at the level of complete cases.
  • Experimental protocol: The study evaluates random forest, XGBoost, logistic regression, and SVM while optimizing configurations separately for dataset, bucketing, and encoding combinations.TPE and three-fold cross-validation select configurations using mean AUC on training prefixes.

5.3 Results: accuracy and earliness

Across 24 tasks, XGBoost and RF are generally the strongest classifiers, while aggregation encodings and simpler bucketing approaches perform competitively. Performance often increases with prefix length, but concept drift and dataset characteristics produce important exceptions.

  • Classifier comparison: XGBoost achieves the highest AUC in 15 of 24 datasets and the highest F-score in 11, while RF leads AUC in 11 and F-score in 14.Logit leads AUC in 7 and F-score in 6 datasets; SVM generally performs worse than the other classifiers.
  • Classifier comparison: XGBoost has the best average rank at around 1.8, but its difference from RF and logit is not statistically significant; SVM is significantly worse than XGBoost and RF.The comparison uses a Nemenyi test over the best AUC for each classifier and dataset at p < .05.
  • Bucketing and encoding: Single aggregation achieves the best AUC in 10 of 24 datasets, followed by prefix aggregation in 8, while last-state encodings generally underperform aggregation counterparts.The Nemenyi comparison places prefix aggregation slightly ahead of single aggregation, with several differences not statistically significant.
  • Earliness: Aggregation methods often reach AUC = 1 at longer prefixes, although bpic2012 and sepsis show declines at larger prefix sizes.Evaluation uses prefixes of exactly each length, so the number of cases decreases as prefix length increases.
  • Bucketing and encoding: When optimized cluster counts are small, cluster aggregation behaves similarly to single aggregation; with more clusters, its accuracy can drop relative to single aggregation.Examples include bpic2012 1 and hospital 2.
  • Earliness: Aggregation encoding can outperform last-state encoding as prefixes grow, whereas last-state encoding retains limited information; bpic2011 4 is an exception linked to concept drift and overfitting.In bpic2011 4, aggregation produced a more complex classifier that memorized training data, while concept drift affected aggregated features more strongly.
  • Bucketing and encoding: Index-based encoding leads in only 4 datasets and often does not outperform lossy encodings, suggesting that event-order information is not consistently useful for these tasks.The authors also report that KNN is not among the top-performing methods and conclude that few or single classifiers with more traces are preferable.

5.4 Results: time performance

Execution times differ mainly between offline training strategies, while online processing is generally fast across methods. KNN has negligible offline work but variable online cost, and RF is usually slower online than the other classifiers.

  • Classifier timing: Logistic regression is the fastest classifier offline, while logit, SVM, and XGBoost have comparable online times and RF is usually slower.The ordering of offline times among the remaining classifiers varies across small and large datasets.
  • Offline performance: KNN has almost negligible offline time because its classifiers are built at runtime, whereas other methods require offline model construction.KNN offline measurements still include prefix-log construction and encoded historical-prefix setup.
  • Offline performance: State-based bucketing is usually fastest offline, followed by prefix-length or clustering-based bucketing, while single-bucket training is slowest.Training multiple smaller classifiers can take less time than training a few larger classifiers on more data.
  • Online performance: Most methods process an event online in less than 100 milliseconds, with hospital datasets requiring around 0.3–0.4 seconds.Prefix-index methods often take about twice as long as other methods, while KNN varies by dataset.
  • Classifier timing: For RF, single classifiers perform relatively better offline; for logit, most methods have comparable offline times, while index-based encoding and clustering are slower.For SVM, single last-state is often much slower, and index-based encoding remains the slowest technique.

5.5 Results: gap-based filtering

Gap-based filtering substantially reduces offline execution time while preserving overall AUC, but has little effect on online execution time.

  • Experimental setup: The experiment evaluates four selected methods and focuses on methods with high offline costs or high overall AUC, across six representative datasets.
  • Gap-based filtering: Gap size g=3 improves offline execution times by about 2–3 times, while g=5 usually improves them by around 3–4 times versus no filtering.For single agg on bpic2017 2, g=5 reduces offline time to about 30 minutes instead of 2 hours.
  • Gap-based filtering: Overall AUC remains at the same level under gap-based filtering and sometimes increases compared with no filtering.
  • Gap-based filtering: Gap-based filtering produces only a marginal positive effect on online execution times, which usually remain at the same level as without filtering.

5.6 Results: categorical domain filtering

Filtering dynamic categorical attribute levels creates execution-time savings, especially for prefix index, but can reduce AUC; intermediate filtering proportions offer practical tradeoffs.

  • Dynamic categorical domain filtering: Offline and online execution times tend to increase linearly as the proportion of retained dynamic categorical levels increases.
  • Dynamic categorical domain filtering: Prefix index benefits most from dynamic-level filtering because its feature vector grows more rapidly as additional levels are included.The vector contains one feature per level per event.
  • Dynamic categorical domain filtering: Filtering dynamic categorical levels negatively affects overall AUC, although reasonable tradeoffs between accuracy and execution time remain possible.
  • Dynamic categorical domain filtering: Using 50% of the levels for single agg on hospital 2 leaves AUC almost unaffected, reduces training time by more than 30 minutes, and halves online execution time.
  • Static categorical domain filtering: Filtering static categorical domains yields only marginal execution-time improvements because each static level occurs in the feature vector only once.

6 THREATS TO VALIDITY

The study’s validity is constrained by possible literature-selection bias and by limited experimental coverage, including restricted methods, algorithms, tasks, and datasets.

  • Threats to validity: The literature review may contain selection bias from Google Scholar updates, subjective inclusion decisions, and primary-versus-subsumed study classification.The authors mitigate these risks through reproducible procedures, public exclusion records, independent assessment by two authors, and third-author mediation.
  • Threats to validity: The experiments test only one clustering method, one state abstraction for state-based bucketing, and four classification algorithms.Untested combinations may outperform the configurations used in the evaluation.
  • Threats to validity: Findings may not generalize fully because the evaluation covers 24 prediction tasks constructed from nine event logs.The logs span different application fields, but other datasets or preprocessing techniques could produce different results.

7 CONCLUSION

The study reviews and categorizes outcome-oriented predictive monitoring methods, then benchmarks them in a unified framework. The benchmark favors lossy aggregation encodings, highlights the impact of concept drift, and identifies gaps for future work.

  • Contribution: The study’s systematic review identified 14 studies on case-outcome prediction, including seven primary studies with distinct contributions.
  • Contribution: The authors performed a unified comparative evaluation of 11 techniques across 24 predictive monitoring tasks from nine real-life event logs.All selected techniques were implemented in a publicly available framework extensible with additional datasets and methods.
  • Benchmark findings: The benchmark found the most reliable and accurate AUC results with lossy aggregation encodings, such as frequencies of performed activities rather than ordered activities.
  • Benchmark findings: Aggregation encoding represents prefixes of any length with the same number of features, allowing one classifier to learn from all prefix traces.This finding conflicts with the existing view that lossless index-based encoding is superior.
  • Benchmark findings: Concept drifts in data attributes from two datasets significantly affected the performance of all tested methods.
  • Future work: Existing methods largely exclude textual attributes, use mainly intra-case features, and include only one inter-case feature, leaving these areas open for future work.The study also identifies LSTMs as a direction for outcome prediction and automated feature derivation.

APPENDIX

The appendix collects supplementary distributions, performance results, concept-drift analyses, and execution-time measurements for the benchmarked methods.

  • It reports case-length histograms for positive and negative outcome classes and bucket-size distributions for different bucketing methods.
  • It provides best cluster and neighbor settings identified for the evaluated clustering and KNN approaches.
  • It lists overall AUC and F-score values for random forest, logistic regression, and SVM classifiers.
  • It presents AUC across prefix lengths for XGBoost using all compared methods and using long traces only.
  • It examines concept drift in the bpic2011 4 and sepsis 1 event logs.
  • It reports offline and online execution times for multiple classifiers, plus execution times and AUC when filtering static categorical attribute domains with XGBoost.
Loading 1707.06766v4…