PaperlayerView exact revision on arXiv

Source-linked AI summary

Predicting Process Behaviour using Deep Learning

Joerg Evermann, Jana-Rebecca Rehse, Peter Fettke

arXiv:1612.04600v2cs.LGstat.ML

TL;DR

Predicting future business-process behaviour matters for business process management, but prior process prediction largely relied on explicit process models. This paper applies deep recurrent neural networks to next-event prediction and reports cross-validated precision above 80% on many problems, demonstrating feasibility and usefulness.

  • Problem

    Predicting future business-process behaviour is important for business process management, while process prediction has largely relied on explicit process models.

  • Method

    The paper applies deep learning with recurrent neural networks to predict the next event from process traces and associated information.

  • Results

    Cross-validated precision exceeded 80% on many problems, with results surpassing or approaching the state of the art.

  • Takeaways & Limitations

    The results demonstrate the feasibility and usefulness of process prediction without relying on explicit process models.

  • Takeaways & Limitations

    A main criticism of neural networks is that they provide no explicit, human-readable representation of the process.

Abstract

from arXiv · show

Predicting business process behaviour is an important aspect of business process management. Motivated by research in natural language processing, this paper describes an application of deep learning with recurrent neural networks to the problem of predicting the next event in a business process. This is both a novel method in process prediction, which has largely relied on explicit process models, and also a novel application of deep learning methods. The approach is evaluated on two real datasets and our results surpass the state-of-the-art in prediction precision.

1. Introduction

The paper introduces deep recurrent neural networks for predicting the next event in business processes, adapting next-word prediction ideas from NLP. It reports improved precision, avoids reliance on explicit process models, and demonstrates feasibility for Smart BPM applications.

  • Contribution: The approach applies historical process data and AI learning technology to real-time case management, execution monitoring, and analysis.The paper presents this as opening new perspectives for process management and Smart BPM.
  • Motivation: The approach targets practical uses including estimating resolution or production-completion time and identifying likely compliance violations.These use cases connect process prediction with customer service, production planning, utilization, and business-risk mitigation.
  • Contribution: The paper introduces a deep-learning approach using recurrent neural networks to predict the next event in a business process.The approach interprets process event logs as sentences, traces as sentences, and events as words, following NLP next-word prediction ideas.
  • Contribution: The method improves state-of-the-art next-event prediction precision.The authors describe their results as considerably better in precision on next-event prediction.
  • Contribution: Deep-learning models can perform as well as explicit process models while representing process structure only implicitly.This removes the need for an explicit process model for prediction.
  • Contribution: The paper extends earlier work with advanced neural-network cells, cross-validation, parameter studies, case-remainder prediction, state visualization, timing information, and NLP comparisons.These extensions support empirical exploration of efficiency and effectiveness.

2. Related Work

Prior process-prediction methods used explicit state-transition, HMM, PFA, sequence-mining, regression, or decision-tree representations. This paper instead uses an RNN with implicit process representation, seeking fewer restrictive assumptions while targeting next-event prediction.

  • Existing approaches: Existing process-prediction research covers diverse techniques and objectives, including next-event and remaining-time prediction.The related approaches use process logs, trace prefixes, state representations, and predictive models for operational decision support.
  • Existing approaches: Five prior next-event approaches used explicit state-transition, HMM, or PFA process representations.The paper contrasts these explicit representations with its deep-learning approach.
  • Existing approaches: The MSA method builds a state-transition matrix from observed prefixes and predicts from the most similar observed case.Similarity is computed with string edit distance when a running case reaches an unseen state.
  • Existing approaches: MSA reported next-event accuracies of up to 25% and 70% on two telecommunications datasets.These values correspond to the two datasets evaluated by Le et al.
  • Proposed distinction: The paper’s RNN predicts next events from event sequences and resource information, representing processes implicitly rather than through state-transition, PFA, or HMM models.Neural networks are presented as suitable for nonlinear relationships and less dependent on ex-ante assumptions about model form.
  • Proposed distinction: The authors characterize their approach as requiring less restrictive assumptions because its model is built implicitly rather than explicitly.They present this as an innovative approach to process prediction.

3. Deep Learning for Process Prediction

The paper adapts recurrent neural networks, particularly LSTM cells, to predict process events by treating event logs like language sequences. The architecture maintains sequence state and produces next-event probabilities through an embedded input and softmax output.

  • Recurrent Neural Networks (RNN): RNN recurrence feeds information back into cells, preserving state across sequence positions through an unrolled computational graph.
  • A recurrent neural network predicts the next process event from a sequence of observed events.
  • The central analogy represents event logs as text, traces as sentences, and events as words.
  • Long Short Term Memory (LSTM): LSTM cells regulate memory with forget, input, and output gates that discard prior information, update cell state, and expose selected state values.
  • Long Short Term Memory (LSTM): The LSTM state update combines retained prior information with newly computed candidate information, while the output gate produces the cell output.
  • Long Short Term Memory (LSTM): Peepholes let LSTM gates inspect prior or current cell state, although evaluated LSTM modifications may not significantly improve performance.
  • The network embeds categorical inputs as trainable numeric vectors and uses a softmax output layer to select the most probable next item.
  • Language and Process: The paper reports that process event logs show exponential information decline, so both Markov-based models and RNNs can model them.

4. Implementation

The implementation uses a two-layer LSTM RNN with configurable embedding dimensionality and unrolled sequence length. Training uses TensorFlow, mini-batches, gradient-based optimization, dropout, and scheduled learning-rate reduction.

  • The approach is implemented in TensorFlow, supports parallel, cluster, and GPU platforms, and has code, data, and complete results available from the corresponding author.
  • The implemented network has two hidden RNN layers using LSTM cells, with embedding dimensionality and unrolled steps as key design choices.
  • Embedding dimensionality affects separation of words in the representation space and may improve prediction at greater computational cost.
  • The experimental baselines are 125 for embedding dimensionality and 20 for unrolled steps, with both parameters varied for evaluation.
  • More unrolled steps allow earlier process events to influence prediction, but increase computational effort.
  • General neural-network parameters are treated as less important for performance than network size and largely independent in their effects.
  • Training uses batches of size 20, runs for 100 epochs, and reduces the learning rate from 1 by 0.75 each epoch after epoch 50.
  • Dropout randomly removes cells during training to reduce overfitting, using a probability of 0.2 per cell.

5. Experimental Method

The evaluation uses real BPI Challenge 2012 and 2013 process datasets, varied event representations and organizational information, and 10-fold cross-validation to assess RNN prediction precision and generalizability.

  • Comparison: Comparison with prior methods is constrained because most related authors did not provide software or data for comparative evaluation.The paper uses publicly available data and notes that Breuker et al. provide the principal directly comparable implementation.
  • Datasets: The experiments use real BPI 2012 loan-application data and BPI 2013 IT incident and problem-management data.BPI 2012 contains 13,087 traces; the BPI 2013 datasets contain 7,553 and 2,300 traces.
  • Datasets: BPI 2012 is evaluated both by subprocess and as a combined dataset, using completion-only events and, for selected data, all lifecycle events.The all-events condition includes Start, Schedule, and Complete transitions; only the W subset contains non-completion events.
  • Input representation: Composite event representations concatenate activity names, lifecycle transitions, and resources, increasing vocabulary size while enabling resource prediction.For BPI 2013, organizational groups replace individual resources because the number of distinct resources is very large.
  • Evaluation method: The study addresses stochastic variation through 10-fold cross-validation and reports mean and standard deviation for training and validation precision.Validation precision assesses generalizability and potential overfitting to the training sample.
  • Evaluation method: Prediction precision is measured as the proportion of correct predictions, with training and validation results summarized across all 10 folds.The evaluation compares training precision by epoch with validation precision by epoch.

6. Experimental Results

Across datasets, 100 epochs generally provide stable results, with convergence speed varying by vocabulary size and dataset. Validation precision usually tracks training precision at a lower level, while comparisons with prior work are dataset-dependent.

  • Convergence: 100 training epochs are sufficient for optimal and stable results across the evaluated datasets.Training precision is averaged across 10 training folds.
  • Convergence: Datasets with smaller vocabularies converge faster than BPI 2012 W and BPI 2013 Incident, which include resource or organizational-group information.The latter datasets consequently have larger vocabularies.
  • Convergence: Adjusting the training rate at epoch 50 suggests that a dynamic training rate helps prevent suboptimal convergence.The validation curve generally follows training precision at a lower level.
  • Comparative results: For BPI 2013 Problems, training precision improves over the comparator but validation precision is lower, while BPI 2012 W performs poorly.The paper relates these outcomes to expectations based on mutual information.
  • Generalization: Validation precision is generally within ±0.05 of training precision, indicating little overfitting in these evaluations.Validation standard deviations are about an order of magnitude larger because validation samples contain one tenth as many cases.
  • Comparative results: Adding all events significantly improves prediction for BPI 2012 W compared with completion events only because Start events regularly precede corresponding Complete events.This regularity provides a predictable event pattern.

6.2. The Effect of Resource Information

Resource or organizational-group information can improve training precision and enable prediction of the next event’s resource, but its validation effects vary and may reflect overfitting or irregular resource assignments.

  • Validation behavior: Validation precision does not consistently follow training improvements and may improve less, suggesting a risk of overfitting when resource information is included.The validation effect differs across datasets.
  • Predictor and predictand: Adding resource information to both predictor and predictand enables prediction of the next event’s activity and associated resource.The trade-off is a larger vocabulary, which can reduce prediction precision.
  • Predictor information: Including organizational information in the predictor improves training precision for all datasets by up to 0.07.This condition adds resource or organizational-group information only to the predictor.
  • Dataset dependence: Including organizational information in both predictor and predictand affects datasets differently, with validation precision dropping significantly for the reported BPI 2013 and BPI 2012 datasets.Training precision decreases for BPI 2013 Incident but increases for BPI 2013 Problem.
  • Dataset dependence: For BPI 2012 W with completion events only, training precision falls to .313 and validation precision to .208 when resource information is included in both fields.The assigned resources lack underlying regularity, impairing prediction performance.
  • Dataset dependence: For BPI 2012 W with all events, matching resources on completion and start events lead to improved performance.The improvement is associated with regularity in the resource assignments.

6.3. Predicting Duration of Activities

The RNN is extended to predict activity duration by encoding each minute of an activity as a repeated activity token. On BPI 2012 W, this produces substantially higher validation precision.

  • Duration representation: The duration-prediction representation quantizes a trace into minutes and encodes the current activity at each minute.An activity A lasting three minutes becomes the sequence AAA.
  • Duration representation: Only BPI 2012 W provides both start and completion events needed to determine activity duration.The dataset contains long-running cases, making idle-time encoding potentially produce very long sequences.
  • Results: 0.942 validation precision is achieved for duration prediction, with SD=0.027, significantly exceeding 0.832 without duration information.The increase is attributed to longer sequences of identical words that make the following word easier to predict.
  • Results: The duration experiment demonstrates that repeated activity tokens can make next-event prediction easier in this dataset.This conclusion is specific to the BPI 2012 W duration-prediction setting.

6.4. The Effect of Embedding Space Dimensionality

Embedding dimensionality has little effect above a dataset’s vocabulary size, but smaller spaces reduce precision, especially for large-vocabulary datasets. Increasing dimensionality to 500 causes overfitting, while the 125-dimensional baseline is near-optimal without significant overfitting.

  • Reducing embedding dimensionality generally harms prediction precision, but the effect is negligible when dimensions exceed vocabulary size.The reduction becomes more pronounced at 8 dimensions.
  • BPI 2012 and BPI 2013 datasets show no marked precision reduction when dimensionality decreases from 64 to 32 and 16, but reductions are more pronounced at 8.This comparison uses datasets with and without organizational information.
  • Datasets with organizational information and larger vocabularies show stronger precision losses at smaller embedding dimensions.Examples include BPI 2013 datasets with 3133 and 64 unique activity–organizational-group combinations and BPI 2012 with 877 combinations.
  • 500 embedding dimensions produce significant overfitting, particularly for datasets with large vocabularies.Training precision rises while validation precision is lower and decreases as dimensionality increases.
  • The 125-dimensional baseline is close to the optimum validation precision across the evaluated datasets.It also avoids significant overfitting when other parameters remain unchanged.

6.5. The Effect of the Number of Unrolled Steps

Changing the number of unrolled steps from the baseline to 10 or 5 has little overall effect on training or validation precision. This suggests that the evaluated processes contain few or no long-term dependencies.

  • Neither training nor validation precision is significantly affected by the number of unrolled steps.The experiments compare 10 and 5 unrolled steps while keeping other parameters unchanged.
  • The BPI 2013 Problem dataset shows a minor pattern in which training performance decreases while validation performance increases with more unrolled steps.The effect is more pronounced when organizational-group information is included.
  • The results indicate few to no long-term dependencies in the evaluated processes.This interpretation is consistent with the mutual-information analysis referenced by the authors.

6.6. Interpreting the RNN

The paper addresses the limited interpretability of neural networks by examining generated traces and hidden-state activations. These analyses indicate that the RNN learns relevant event-log features and can reproduce realistic process behaviour.

  • Neural networks lack an explicit human-understandable model because learning optimizes thousands or millions of floating-point parameters.The paper presents two ways for users to inspect knowledge encoded in the network.
  • Hallucinations: Hallucinations generate process traces from a short seed sequence by selecting either the highest-probability event or sampling from output probabilities.The paper evaluates both output strategies on the full BPI 2012 and BPI 2013 datasets.
  • Hallucinations: Probabilistic output sampling produces more realistic traces than highest-probability selection, whose outputs often contain long uniform sequences uncharacteristic of the input logs.The authors report similar results for the other event logs.
  • Hallucinations: Generated hallucinations reproduce realistic process traces, indicating that the RNN learned important features of the training event log.The authors use this reproduction to validate the trained network and support the usefulness of the RNN approach.
  • Trace completion: Hallucinations can predict case remainders by continuing a trace prefix until an end-of-case indicator and comparing the result with the actual continuation using normalized Damerau–Levenshtein distance.The distance ranges from 0 to 1, with the evaluation using prefixes of length 5.
  • Applications: Hallucinations may support process-mining applications such as improving event-log completeness when algorithms benefit from better or larger logs.This is presented as a potential application of the generated traces.
  • Hidden-state visualization: The authors find no significant or obvious clustering of events in their two-dimensional t-SNE embedding plots.The plots are created from embedding matrices exported after training.
  • Hidden-state visualization: LSTMVis visualizes hidden-state activations to connect activation patterns with event sequences, such as declined loan applications ending in ADECLINED followed by [EOC].Other cells represent additional events and event sequences in the BPI 2012 dataset.

7. Discussion and Conclusion

The paper introduces recurrent deep learning for process-event prediction without requiring explicit process models, achieving strong reported precision while exposing trade-offs in generalizability, simplicity, and input representation.

  • Contribution: The paper introduced deep learning with recurrent neural networks for predicting the next event in business processes.The approach brings historic process data together with AI learning technology for process prediction.
  • Contribution: The approach does not rely on explicit process models, whose abstraction and parsimony can impair predictive performance.Deep learning networks use larger parameter spaces and non-linearity to avoid intentionally imperfect intermediate representations between event logs and prediction.
  • Results: Cross-validated precision exceeded 80% on many problems, with validation precision close to or above 0.8 in numerous results.The reported results surpassed or approached the state of the art in prediction precision.
  • Comparison: RNNs support prediction from trace prefixes of arbitrary length and surpassed PFA performance in many cases, although they were not always superior.The paper chose RNNs because they fit the sequential structure of process event logs.
  • Limitations: The approach is constrained by overfitting risks and trade-offs between prediction precision, generalizability, and model simplicity.Cross-validation is described as essential for assessing overfitting, generalizability, and model size together with predictive performance.
  • Limitations: The input encoding is most feasible when attributes have few unique values; large resource sets and numerical attributes can limit applicability or cause information loss.An alternative encoding strategy and methods for identifying useful event- and case-level predictors are identified as future work.
Loading 1612.04600v2…