Source-linked AI summary
Automated Event Log Generation from Unstructured Text Using Finetuned LLMs
Maximilian Seeth, Gabriel Marques Tavares, Daniel Schuster
TL;DR
Process mining depends on structured event logs that are costly to create, leaving textual organizational resources underused. The paper develops a synthetic text-to-log dataset and finetunes LLMs as translators from natural-language process descriptions to event logs. Finetuning substantially improves syntactic validity, reduces hallucinations, and supports process discovery compared with prompting-only approaches.
Problem
Process mining requires well-structured event logs, but their expert-led preparation is laborious and natural-language process descriptions are not reliably mapped into usable logs.
Method
The paper creates synthetic text-to-log data, finetunes relatively small open-weight LLMs, and evaluates generated event data at trace and process-model levels.
Results
Supervised finetuning substantially improves syntactic validity, reduces hallucinations, and yields event data supporting meaningful process discovery with established mining algorithms.
Takeaways & Limitations
The findings highlight supervised finetuning as a practical and scalable alternative to prompt-only or process-model-only approaches for generating event logs from unstructured data.
Takeaways & Limitations
The evaluation uses synthetic corpora, and the output is currently constrained to the XES standard and its case-centric paradigm.
Abstract
from arXiv · showhide
Process mining (PM) provides a powerful framework for discovering and optimizing operational processes from event data. However, the efficacy of PM techniques is strictly predicated on the availability of structured event logs. Thus far, event logs have often been laboriously created by domain and process mining experts. This costly effort causes large portions of organizational knowledge, including incident tickets, manuals, and textual reports, to remain underutilized. We address this bottleneck by investigating the efficacy of Large Language Models (LLMs) as automated data translators. We propose a scalable framework that leverages LLMs as data translators to bridge the gap between unstructured textual resources and structured event data. We finetune LLMs on a newly created text-to-log dataset, demonstrating that the resulting models can extract high-fidelity event logs from unstructured resources. Our results show that this finetuning approach outperforms few-shot or zero-shot prompting by a large amount, highlighting finetuning as a necessary pre-condition for generating reliable event data. We conclude that our method provides a promising pipeline for making previously unused data available to process mining ecosystems, effectively expanding the possibilities of using PM to further investigate organizational workflows.
1 Introduction
The paper addresses the costly dependence of process mining on expert-created event logs by evaluating LLMs as translators from natural-language process descriptions to structured, usable event data. It develops synthetic text-to-log data, finetunes open-weight LLMs, and evaluates generated logs at trace and process levels.
- Event-log preparation and extraction require substantial time and effort, leaving incident tickets, manuals, and reports underutilized.
- The study evaluates whether LLMs can map natural-language process descriptions into coherent, schema-compliant event logs for process mining.
- The proposed approach delegates process modeling to established algorithms while using LLMs to provide them with appropriate structured data.
- Supervised finetuning substantially improves accurate and well-formed event-data generation compared with zero-shot and few-shot prompting.
- Because suitable paired data are unavailable, the authors create synthetic text-to-log datasets by transforming XES-formatted traces with frontier foundation models.
- The evaluation framework assesses generated logs at both individual-trace and process-level perspectives.
2 Related Work
Prior process-mining research uses language models for process-model generation or event extraction, but differs from this work in model type, supervision, data pairing, and evaluation scope. This paper emphasizes decoder-based open-weight LLMs, supervised finetuning, synthetic text-to-log data, and evaluation beyond individual traces.
- Many prompting-based process-mining approaches generate complete process models directly from unstructured documents rather than event logs.
- Earlier encoder-based approaches derive event representations from text and synthesize XES logs, but do not finetune for downstream prediction or compare against gold standards.
- Its pipeline generates synthetic texts from real-world event logs, uses different LLMs for text generation and event-data extraction, and evaluates results against original logs.
- Unlike related work focused only on individual traces, the authors also evaluate holistic process representations using discovery algorithms including the Inductive Miner and Heuristics Miner.
- The study treats supervised finetuning as its primary focus and compares three open-weight LLMs under zero-shot and few-shot prompting strategies.
- The paper differs by using decoder-based LLMs as bridges from varied natural-language texts to structured event logs, rather than predicting continuations of already structured event streams.
3 Synthetic Data
The authors construct paired synthetic text-and-event-log datasets from traffic-fine and sepsis-treatment processes, then evaluate generated texts for diversity, semantic variability, faithfulness, and temporal clarity.
- Existing datasets lack pairings between natural-language process descriptions and structured event logs, motivating the creation of specialized synthetic datasets.
- The datasets derive synthetic police reports from a traffic-fine log and discharge summaries from a sepsis-treatment log.The traffic-fine data cover invoice issuance through payment collection, while the sepsis data cover patient registration, treatment, and discharge.
- Three frontier foundation models generate texts using varied personas, prompts, and stochastic decoding to increase textual diversity and realism.Five personas, three prompts, temperatures from 0.6 to 0.9, and top-p 0.9 were used.
- Median police-report length remains below 500 tokens, while Gpt-5.1-2025-11-13 produces sepsis summaries with median length above 800 tokens.Qwen3-max-2025-09-23 produces summaries less than half that size, with Mistral-medium-2508 between the two models.
- For sepsis summaries, Gpt-5.1-2025-11-13 reaches 5-gram ratio R_n = 0.75 versus 0.57 for Mistral-medium-2508.Higher n-gram ratios indicate greater lexical diversity, and all reports show ratios increasing with n.
- Mistral-medium-2508 consistently yields the greatest semantic differences, although model results remain close overall.The authors compute average pairwise Euclidean distances between document embeddings; the sepsis summaries are nevertheless more similar to one another than the police reports.
- Annotators assigned high faithfulness and temporal-clarity scores to synthetic police reports, with μ >= 0.95 and raw agreement of 0.900 and 0.967.Cohen’s Kappa values were low because highly skewed defect base rates create the Cohen’s Kappa paradox.
4 Method
The method compares zero-shot and few-shot prompting with supervised finetuning for translating natural-language reports into XES-formatted event traces.
- Zero-shot and few-shot prompting generate outputs from the language models under an identical task instruction for comparability.Few-shot prompting adds a randomly sampled natural-language text and XES-trace demonstration.
- Randomly sampled few-shot demonstrations reduce bias from repeated exposure to a fixed example and lower the risk of overfitting.
- Supervised finetuning uses reports as inputs and corresponding XES-formatted traces as targets.The police-report pairs are split into training, development, and test sets at 85%, 5%, and 10%.
- Finetuning is applied to Llama3.1-8B-Instruct, Ministral-8B-Instruct-2410, and Qwen2.5-7B-Instruct, selected for potential local deployment by process-mining practitioners.
- The finetuning setup uses Low-Rank Adaptation with rank 16 and computes loss only over assistant output tokens.Prompt tokens are masked during training, and examples include a system prompt, instruction, one-shot demonstration, and input-target pair.
5 Experimental Setup and Evaluation
The evaluation tests LLM-generated XES logs across trace validity, similarity, variants, hallucinations, and process-model quality, comparing zero-shot, one-shot, and finetuned settings. Finetuning generally improves structural and behavioral alignment, including under cross-domain and held-out-generator tests.
- Trace-Level Evaluation: One-shot demonstrations substantially improve the generation of well-formed XES structures from police reports.
- Trace-Level Evaluation: 430 of 434 traces generated by finetuned Llama3.1-8B-Instruct are valid, while Qwen2.5-7B-Instruct produces two well-formed structures even zero-shot.
- Trace-Level Evaluation: Finetuning reduces variant-count deviations: Llama3.1-8B-Instruct and Ministral-8B-Instruct-2410 generate 46 and 43 variants versus 43 in the original log.
- Trace-Level Evaluation: Finetuned Llama3.1-8B-Instruct and Ministral-8B-Instruct-2410 closely match original event ordering and lengths, with Kendall’s Tau means of 0.99 for both.Their Levenshtein distance means are 0.99 and 1.0, whereas finetuned Qwen2.5-7B-Instruct scores 0.74 on Levenshtein distance and 0.42 on Kendall’s Tau.
- Process-Level Evaluation: Finetuning improves process-model quality: inductive F1 reaches 0.51, 0.44, and 0.43 for Llama3.1, Ministral, and Qwen2.5, respectively, versus 0.58 for real logs.Held-out-generator F1 rises by 12–17 percentage points for inductively mined models and by 36 and 40 percentage points for heuristically mined models.
6 Discussion and Limitations
The framework uses synthetic text–log pairs to make finetuning scalable, but its evidence is bounded by synthetic-report structure and XES’s case-centric representation. The authors identify support for object-centric data and real reports as important extensions.
- Limitations: Synthetic police reports differ from real police documentation, which often combines form-like and narrative structures.The authors believe the semantic extraction capabilities required may remain largely invariant despite these structural differences.
- Discussion: Synthetic corpora provide a scalable source of finetuning data for specialized process-mining contexts where real-world data are inaccessible.The authors treat synthetic data generation as a standalone contribution of the pipeline.
- Limitations: The current output is constrained to the XES standard and its case-centric paradigm.This representation does not yet account for the many-to-many relationships of object-oriented structures such as OCEL.
- Future directions: Extending the framework to object-centric event data remains a priority for future iterations.The authors also identify validation on real, non-synthetic reports as an important next step.
7 Conclusion
The paper investigates open-weight LLMs as translators from unstructured process descriptions to executable event logs, using synthetic text–log pairs for supervised finetuning. Evaluation at trace and process-model levels shows improved event-data quality and supports future expansion across domains, modalities, and representations.
- Conclusion: The framework investigates open-weight LLMs as direct translators from unstructured process descriptions to executable event logs.It targets standards-compliant event logs and process models for process mining.
- Conclusion: Synthetic text–log pairs generated from real-world process logs serve as training data for supervised finetuning.The approach is designed to make paired data available for training event-log translators.
- Conclusion: On average, supervised finetuning improves the quality of event logs used to derive Inductive and Heuristics process models, including across text distributions or domains.The comparison is summarized in Fig. 8.
- Conclusion: Finetuning substantially improves syntactic validity, reduces hallucinations, and yields data that supports meaningful process discovery.These findings are reported across evaluations at both the trace and process-model levels.
- Conclusion: The results position supervised finetuning as a practical and scalable alternative to prompt-only or process-model-only approaches.The authors connect this to better exploitation of previously untapped textual process descriptions.
- Future directions: Future work includes training cross-domain data translators, using reinforcement learning with verifiable rewards, integrating richer modalities, and exploring object-centric representations.Validation on real, non-synthetic reports is also identified as important for fully automated process-data generation.