PaperlayerView exact revision on arXiv

Source-linked AI summary

Survey and cross-benchmark comparison of remaining time prediction methods in business process monitoring

Ilya Verenich, Marlon Dumas, Marcello La Rosa, Fabrizio Maggi, Irene Teinemaa

arXiv:1805.02896v2cs.AIcs.LG

TL;DR

The paper addresses unclear relative merits among remaining-time prediction methods caused by differing experimental setups, datasets, measures, and baselines. It conducts a systematic review and taxonomy, then cross-benchmarks 16 methods on 16 real-life event logs. LSTM networks achieve the most accurate results overall, while simpler methods can perform comparably on average.

  • Problem

    The relative merits of remaining-time prediction methods remain unclear because prior studies use differing experimental setups, datasets, evaluation measures, and baselines.

  • Method

    The paper performs a systematic literature review and taxonomy, then evaluates 16 methods using a unified benchmark across 16 real-life event logs.

  • Results

    LSTM networks obtain the most accurate results overall, while canonical bucketing-and-encoding combinations do not differ significantly on average.

  • Takeaways & Limitations

    When simpler algorithms perform on par with more complex ones, choosing the simpler method is often preferable, although event-log properties can favor specific combinations.

  • Takeaways & Limitations

    Generalizability is limited because the experiments used only 16 event logs, and results may differ with other datasets or preprocessing techniques.

Abstract

from arXiv · show

Predictive business process monitoring methods exploit historical process execution logs to generate predictions about running instances (called cases) of a business process, such as the prediction of the outcome, next activity or remaining cycle time of a given process case. These insights could be used to support operational managers in taking remedial actions as business processes unfold, e.g. shifting resources from one case onto another to ensure this latter is completed on time. A number of methods to tackle the remaining cycle time prediction problem have been proposed in the literature. However, due to differences in their experimental setup, choice of datasets, evaluation measures and baselines, the relative merits of each method remain unclear. This article presents a systematic literature review and taxonomy of methods for remaining time prediction in the context of business processes, as well as a cross-benchmark comparison of 16 such methods based on 16 real-life datasets originating from different industry domains.

1 INTRODUCTION

Predictive business process monitoring uses historical event logs to predict properties of ongoing cases, including remaining cycle time, to support operational decisions. This paper focuses on remaining-time prediction and contributes a systematic review, taxonomy, and cross-benchmark evaluation.

  • Predictive monitoring uses historical event logs to predict future properties of ongoing process cases.
  • Predictions can alert workers to problematic cases or support resource allocation for cases at risk of deadline violation.
  • The paper focuses on predicting remaining time, completion time, and case duration for ongoing process cases.
  • The study combines a systematic literature review, a taxonomy of existing methods, and a comparative evaluation using real-life event logs.
  • The cross-benchmark evaluates 16 representative methods on 16 real-world event logs from different industry domains.

2 BACKGROUND

The background connects process mining and machine learning to predictive monitoring of event-log data. It defines cases, events, traces, prefixes, supervised learning, model families, and prediction points for remaining-time estimation.

  • Predictive process monitoring combines concepts from process mining and machine learning.
  • Process mining: Enterprise systems record process executions as cases and events, while process mining derives insights from these event logs.
  • Process mining: An event contains an activity name, case identifier, timestamp, and optional event or case attributes.
  • Process mining: A trace is a sequence of events belonging to one case, and an event log is a set of completed traces.
  • Predictive process monitoring: Prefixes expose partial case histories for prediction, with each possible prefix represented in a prefix log.
  • Machine learning: Supervised learning fits predictive models from labeled feature vectors, supporting regression for continuous targets such as cycle time.
  • Machine learning: Discriminative methods model P(Y|X), whereas generative methods model P(X,Y) and derive the conditional distribution for prediction.
  • Predictive process monitoring: Remaining-time prediction uses a partial trace at a prediction point to estimate a future performance measure, such as time until case completion.

3 SEARCH METHODOLOGY

The study uses a systematic literature review to identify and classify remaining-time prediction methods and to compare their performance across real-life process logs. Search, snowballing, and explicit eligibility criteria reduced a broad retrieval set to 24 relevant studies.

  • The review followed a systematic process of defining research questions, constructing searches, applying eligibility criteria, and separating primary from subsumed studies.
  • The research questions address existing methods, classification, evaluation data and domains, tool support, and relative performance.
  • The benchmark sought to classify methods by input data and predictive algorithms while cross-benchmarking them on real-life logs.
  • Study retrieval: The search used specific remaining-time phrases and covered papers published from 2005 through 2017, with the search conducted in March 2018.
  • Study retrieval: 670 unique results were initially retrieved, followed by filtering and snowballing that produced 110 publications for final scope assessment.
  • Eligibility criteria: The exclusion criteria removed studies unrelated to predictive process monitoring, studies without remaining-time targets, and methods lacking essential event-log components.
  • Study retrieval: 24 relevant studies remained after applying the exclusion criteria.

4 ANALYSIS AND CLASSIFICATION OF METHODS

The paper classifies remaining-time prediction methods by input data, process awareness, algorithm family, evaluation data, application domain, and implementation, then surveys their empirical support and workflow. Most methods use machine learning in offline training and online prediction, while real-life validation increasingly relies on public logs.

  • The survey categorizes methods by input data, process awareness, algorithm family, evaluation data and domain, and implementation type.
  • Process awareness: Process-aware methods exploit explicit process representations, including transition systems, query catalogs, queueing models, stochastic Petri nets, and BPMN models.
  • Algorithm families: Non-process-aware approaches commonly use regression trees, random forests, XGBoost, neural networks, trace clustering, or statistical risk indicators.
  • Evaluation data and domains: Most surveyed methods were validated on at least one real-life event log, with logistics and banking represented by 7 studies each, public administration by 5, and customer service by 3.
  • Evaluation data and domains: Many studies increasingly use publicly available logs, while private logs hinder reproducibility because they are inaccessible.
  • Implementation: Nearly half of the methods provide ProM plug-ins, three have standalone Python implementations, and 8 methods lack publicly available implementations.
  • Predictive workflow: Machine-learning methods generally train offline on historical prefixes and generate online predictions by selecting a bucket, encoding the running case, and applying its pretrained model.

5 METHODOLOGICAL FRAMEWORK

The methodological framework abstracts heterogeneous methods into core design choices involving bucketing, prefix encoding, predictors, and process-aware versus discriminative approaches. The benchmark fixes common predictors and encoding choices to reduce comparison bias while preserving selected process-aware alternatives.

  • Because methods require heterogeneous inputs and produce heterogeneous outputs, the benchmark abstracts implementation details and compares their core differences.
  • Bucketing: Machine-learning methods either fit one predictor to the whole event log or multiple predictors to buckets of historical prefixes.
  • Bucketing: The framework considers zero, prefix-length, and cluster bucketing, which determine how prefixes are grouped before model fitting.
  • Prefix encoding: Last-state encoding uses recent events, aggregation encoding summarizes all events with fixed-size statistics, and index-based encoding preserves event positions but increases dimensionality for longer prefixes.
  • Prefix encoding: Tensor encoding represents each prefix as a matrix or higher-dimensional tensor, allowing event data to remain structured rather than flattened into a two-dimensional feature matrix.
  • Predictive algorithms: XGBoost is fixed as the main predictor across compared techniques to eliminate bias from using different predictors, while LSTMs are evaluated separately with zero bucketing and tensor encoding.
  • Taxonomy: The taxonomy frames the benchmark around generative process-aware methods and discriminative approaches, including state-transition, Petri-net, and queueing representations.

6 BENCHMARK

The benchmark evaluates remaining-time prediction methods across 16 real-life event datasets using temporally separated data, multiple accuracy and earliness criteria, and systematic hyperparameter optimization. LSTMs achieve the strongest overall accuracy, while performance varies with dataset characteristics and prefix length.

  • Datasets: The benchmark uses 16 real-life event datasets spanning diverse characteristics and domains, with publicly available logs supporting reproducibility.The datasets include varied case counts, trace diversity, event classes, case lengths and durations, and attribute counts.
  • Data split: Cases are ordered by start time, with the first 80% used for training and the remaining 20% for testing.This temporal split simulates training on historical cases and predicting for later-running cases.
  • Evaluation setup: Predictions are evaluated with MAE, RMSE and MAPE, while earliness is assessed over prefixes up to 20 events.The prefix limit reduces training costs, avoids overrepresenting longer cases, and avoids unreliable estimates for very long prefixes.
  • Evaluation setup: Five-fold cross-validation with grid search selects hyperparameters on the training data before retraining each model on the full training set.Validation uses the same metrics as the test evaluation.
  • Overall results: LSTM-based networks achieve the best accuracy in 13 of 16 datasets, while flow-analysis and index-based encoding with different bucketing schemes lead on one dataset each.The benchmark compares feasible combinations of encoding and bucketing methods alongside other representative predictors.
  • Prefix-length results: MAE generally decreases as cases progress, but later-prefix evaluations can worsen when short completed traces are excluded and only harder longer traces remain.The number of cases evaluated also decreases monotonically with prefix length.
  • Overall results: LSTMs have an average error of 40% of mean case duration, compared with 59% for transition systems; the LSTM range is 0.07–0.56 versus 0.08–0.90 for index-based prefix-length bucketing.The latter method is the second most accurate on average but more volatile across datasets.
  • Dataset-dependent results: Dataset characteristics influence the best encoding and bucketing choice: heterogeneous case lengths favor one bucket, while short or distinctive cases often favor last-state encoding.The authors also observe that aggregating information from all events can provide more signal than using only the latest event.

7 THREATS TO VALIDITY

The study identifies threats from possible literature-selection bias and limited experimental comprehensiveness, while using multiple safeguards to support replicability and consistency.

  • The literature review may suffer from selection bias because academic database rankings can change and inclusion decisions involve subjectivity.The authors documented the review procedure, published inclusion and exclusion decisions, and used independent assessment by two authors with third-author mediation.
  • An open-source framework and publicly documented review decisions were provided to support replication and future extensions.The framework permits additional datasets, sequence-classification methods, and encoding methods to be added.
  • Only one representative setting was used for each technique, limiting the comprehensiveness of the experiments.All relevant methods were tested with one machine-learning algorithm, XGBoost, and one clustering method, k-means.
  • Untested combinations of clustering techniques and predictors may outperform the settings evaluated in this study.This threat follows from evaluating only XGBoost and k-means across the relevant methods.
  • Findings have limited generalizability because experiments used only 16 event logs and may depend on datasets or preprocessing choices.The logs came from different application fields, but results could differ with other datasets or preprocessing techniques.

8 CONCLUSION

The paper surveys remaining-time prediction methods, develops a taxonomy, and evaluates 16 techniques on 16 real-life event logs. LSTM networks were most accurate, while canonical bucketing and encoding combinations did not differ significantly on average.

  • The study identified 24 novel remaining-time prediction studies, of which 10 contained distinct contributions for the proposed taxonomy.The taxonomy classifies methods by input data, process awareness, and algorithm family.
  • Most studies, 7 out of 10, use machine-learning algorithms to train predictive models applied at runtime.Methods were further categorized by trace bucketing and feature-vector encoding, with LSTM networks and flow analysis evaluated separately.
  • The benchmark compared 16 techniques using a unified setup and 16 real-life event logs, with implementations released in a public framework.The framework was intended to support reproducibility of the comparative evaluation.
  • LSTM networks achieved the most accurate benchmark results, possibly because they automatically learn relevant features from trace prefixes.The reported explanation is framed as a possibility rather than an established causal finding.
  • Canonical bucketing and encoding combinations showed no statistically significant average difference, although particular event logs favored some combinations.The conclusion recommends simpler algorithms when they perform on par with more complex alternatives.
Loading 1805.02896v2…