Source-linked AI summary
Automated Discovery of Process Models from Event Logs: Review and Benchmark
Adriano Augusto, Raffaele Conforti, Marlon Dumas, Marcello La Rosa, Fabrizio Maria Maggi, Andrea Marrella, Massimo Mecella, Allar Soo
TL;DR
Automated process discovery has been evaluated inconsistently, limiting comparability and reproducibility across methods. This paper conducts a systematic review and benchmarks representative implementations on diverse real-life logs using an open-source framework and multiple quality metrics. The results reveal method-specific tradeoffs, scalability limitations, and no unique best method across quality dimensions.
Problem
Prior evaluations used different datasets, setups, measures, and baselines, often producing incomparable or unreproducible conclusions.
Method
The paper combines a systematic literature review with a comparative benchmark of seven implementations across 24 real-life event logs and nine quality metrics.
Results
Methods show divergent quality tradeoffs: block-structured methods perform best on fitness or precision and complexity, while Split Miner achieves higher F-score but cannot guarantee soundness.
Takeaways & Limitations
There is no unique winner; method choice depends on the quality dimension, and filtering is necessary before discovery on very complex logs.
Takeaways & Limitations
Using only 24 event logs limits the generalizability of the conclusions, although the benchmark can be rerun with additional datasets.
Abstract
from arXiv · showhide
Process mining allows analysts to exploit logs of historical executions of business processes to extract insights regarding the actual performance of these processes. One of the most widely studied process mining operations is automated process discovery. An automated process discovery method takes as input an event log, and produces as output a business process model that captures the control-flow relations between tasks that are observed in or implied by the event log. Various automated process discovery methods have been proposed in the past two decades, striking different tradeoffs between scalability, accuracy and complexity of the resulting models. However, these methods have been evaluated in an ad-hoc manner, employing different datasets, experimental setups, evaluation measures and baselines, often leading to incomparable conclusions and sometimes unreproducible results due to the use of closed datasets. This article provides a systematic review and comparative evaluation of automated process discovery methods, using an open-source benchmark and covering twelve publicly-available real-life event logs, twelve proprietary real-life event logs, and nine quality metrics. The results highlight gaps and unexplored tradeoffs in the field, including the lack of scalability of some methods and a strong divergence in their performance with respect to the different quality metrics used.
1 INTRODUCTION
Automated process discovery converts event logs into process models, but existing methods struggle to balance fitness, generalization, precision, complexity, and scalability. This paper addresses inconsistent prior evaluation through a systematic review and an open benchmark.
- Event logs contain traces of event sequences associated with individual process cases.
- Automated process discovery takes an event log as input and outputs a business process model capturing observed or implied task-control relations.
- A useful discovered model should reproduce logged behavior, generalize to similar unseen behavior, and avoid generating unrelated traces.
- Existing methods often produce large spaghetti-like models or models with poor fitness, precision, or generalization, making robust quality tradeoffs difficult.
- The study combines a systematic review with a comparative evaluation of seven implementations using 24 real-life logs and nine quality metrics.
- The resulting classified inventory and open-source benchmark support unified, reproducible comparisons of automated process discovery methods.
2 SEARCH PROTOCOL
The review used a structured, replicable search and screening protocol to identify and consolidate research on implemented automated process discovery methods. It ultimately grouped the eligible literature into 35 main algorithm groups and observed changing publication activity over time.
- Research questions: The systematic literature review defined research questions covering discovery methods, model types, semantics, tools, evaluation data, and application domains.
- Search string development: Four search strings combined process or workflow discovery or learning terms, while automated was excluded because it was often absent from papers.
- Study selection: The search used Google Scholar plus six academic databases, followed by inclusion criteria requiring an implemented, evaluated, peer-reviewed English-language study published from 2011 onward.
- Study selection: 2,820 studies remained after automated criteria, 344 after title-and-abstract screening, and 86 after assessment against the first two inclusion criteria.
- Interest in automated process discovery rose over time, with a sharp increase between 2013 and 2014 before later declining toward the yearly average.
3 CLASSIFICATION OF METHODS
The review classifies discovery methods by model type and language, captured semantics, implementation, and evaluation data. Procedural and Petri-net approaches predominate, while the surveyed methods differ in supported constructs and practical availability.
- The classification covers model type and language, procedural semantics, implementation and tool accessibility, and evaluation data and application domain.
- The review identified 35 main method groups and used their consolidated information to answer which automated process discovery methods exist.
- Model type and language: 26 of 35 methods produce procedural models, while six produce Declare-based declarative models, one uses WoMan, and two produce hybrid Petri-net and Declare models.
- Model type and language: Petri nets remain predominant, while newer methods also produce BPMN, Causal nets, State machines, directed acyclic graphs, and Declare models.
- Captured semantics: All 26 procedural methods detect sequence, but only four detect inclusive choice, and three listed methods lack at least one of parallelism, exclusive choice, or loops.
- Implementation: 19 of 35 methods provide ProM plug-ins, reflecting the platform's open-source and portable framework for developing and testing discovery algorithms.
- Evaluation data: Among evaluated methods, 31 of 35 were tested on real-life logs, with some also tested on synthetic or artificial logs.
4 BENCHMARK
The benchmark compares publicly accessible discovery methods producing Petri nets or seamlessly convertible models, using heterogeneous real-life logs and default or optimized parameters. No method dominates every metric: IM, ETM, and SM lead respectively on fitness, precision, and F-score, while scalability and soundness remain important tradeoffs.
- Methods selection: Seven publicly accessible methods producing Petri nets or seamlessly convertible models were selected for comparison.The final set comprised α$, IM, ETM, FO, S-HM6, SM, and HILP.
- Setup and datasets: The benchmark used public real-life logs from healthcare, finance, government, and IT service management, with infrequent behavior filtered from BPIC14, BPIC15, and BPIC17.Filtering was applied because unfiltered comparisons yielded very poor or uncomputable F-scores.
- Benchmark results: The first evaluation found no clear overall winner, with nearly every method exhibiting specific benefits and drawbacks.HILP often produced disconnected or otherwise unusable models, preventing assessment of most accuracy metrics except on PRT5.
- Benchmark results: IM achieved fitness greater than 0.90 twenty times, ETM achieved precision greater than 0.80 nineteen times, and SM outperformed other methods on F-score eighteen times.ETM’s high precision came at the cost of lower fitness, while SM provided high and balanced fitness and precision.
- Benchmark results: SM was fastest 23 of 24 times and discovered a model in less than a second over 19 logs, whereas ETM reached the four-hour timeout on 22 logs.IM and ETM always produced sound, fully block-structured models; ETM and SM also often produced among the smallest models.
- Benchmark results: Hyper-parameter optimization improved F-scores for almost all inputs, and SM achieved the highest optimized F-score over 20 event logs.SM also achieved the best result with default parameters more than half the time.
- Benchmark results: The study concludes that IM, ETM, and SM are most effective when prioritizing fitness, precision, and F-score respectively, but all three struggle with large-scale real-life logs.S-HM6 performed better than FO but produced unsound models, and its precision varied with the input log.
5 DISCUSSION
The review identifies procedural and declarative process-discovery methods, but evaluates only procedural methods because quality-assessment tools constrain cross-stream comparison. Benchmark results reveal substantial tradeoffs across scalability, quality metrics, and execution time.
- The review identifies two main method streams: procedural models and declarative models.
- Quality-assessment tools’ requirements prevented direct comparison between procedural and declarative methods, so the evaluation focused on procedural methods.
- IM, ETM, and SM were the only methods consistently strong across selected fitness, precision, F-score, complexity, and execution-time measures.
- Large and complex event logs exposed scalability limitations, including failures by IM, ETM, and SM on the unfiltered real-life log PRT11.
- No unique winner emerged because the best methods maximized different quality dimensions, including fitness, precision, or F-score.
6 THREATS TO VALIDITY
The study acknowledges threats from literature-review procedures and limits in the experimental evaluation. These include possible selection and extraction bias, restricted method coverage, and a small event-log sample that limits generalizability.
- Literature-review results may be affected by selection bias and inaccuracies in data extraction and analysis.
- The search process was designed for replication, but source-library ranking updates may produce different search results.
- The experimental evaluation covers techniques producing Petri nets or directly translatable models and only studies with available implementations.
- The released open-source toolset supports reproducing the experiments and rerunning the benchmark with additional methods, configurations, or datasets.
- The evaluation uses 24 event logs, which limits the generalizability of its conclusions despite covering real-life logs with varied sizes, features, and domains.
7 RELATED WORK
The paper extends earlier survey and benchmark work by evaluating updated methods with public and proprietary real-life event logs. Its results indicate progress, particularly for SM, ETM, and IM, while differing from earlier rankings.
- De Weerdt et al.’s earlier survey covered 27 approaches and assessed 7, providing the starting point for this study.
- The current benchmark replaces or extends several earlier approaches, including α$, Structured Heuristics Miner, Hybrid ILP Miner, and Evolutionary Tree Miner.
- Unlike the earlier evaluation, this study uses both public and proprietary event logs rather than only artificial logs and closed datasets.
- Earlier work found Heuristics Miner strongest on F-score and simplicity, whereas this benchmark found SM better on F-score and model complexity.
- The newer results identify progress, with SM, ETM, and IM outperforming methods from the previous decade and their extensions.
8 CONCLUSION
The paper combines a systematic review with an open-source benchmark of automated process discovery methods. Its evaluation reveals performance tradeoffs across model structure, quality dimensions, event-log complexity, and the available fitness and precision measures.
- Benchmark and review: The benchmark evaluates existing implementations on 12 public and 12 proprietary real-life event logs using 9 quality metrics.The open-source toolset can be extended with additional methods, logs, and metrics.
- Benchmark and review: The systematic review analyzed 344 relevant papers and found increasing use of BPMN, declarative constraints, and block-structured process models alongside traditional Petri nets.The review also identifies a recent emphasis on block-structured models.
- Empirical findings: Block-structured methods, including Inductive Miner and Evolutionary Tree Miner, achieve the best performance for fitness or precision and complexity.Split Miner achieves higher F-score quality but cannot guarantee soundness, although it can guarantee deadlock-freedom.
- Empirical findings: For very complex event logs, filtering before discovery is necessary because unfiltered models can have precision close to zero.The paper identifies adaptive filtering that auto-tunes to complex logs as a future research direction.
- Limitations and future work: The benchmark exposes a lack of universal fitness and precision measures applicable across Petri-net, BPMN, declarative, and data-driven notations.Developing measures that span these modeling notations is identified as future work.