Source-linked AI summary
Tools and Benchmarks for Automated Log Parsing
Jieming Zhu, Shilin He, Jinyang Liu, Pinjia He, Qi Xie, Zibin Zheng, Michael R. Lyu
TL;DR
Large, unstructured log volumes make manual inspection impractical and complicate downstream analysis. The paper evaluates 13 parsers across 16 datasets, releases reusable tools and benchmarks, and reports accuracy, robustness, efficiency, and Huawei deployment experience. The benchmarks show that most datasets can be parsed above 90% accuracy by at least one parser, while parser performance varies substantially across datasets.
Problem
Huge log volumes and unstructured messages make manual inspection impractical, while handcrafted parsing rules are costly to create and maintain.
Method
The paper implements and evaluates 13 log parsing methods on 16 datasets, using accuracy, robustness, and efficiency benchmarks, and shares an open-source toolkit and datasets.
Results
Most datasets are parsed above 90% accuracy by at least one parser, but parsers show large accuracy variance across datasets and some cannot finish large-volume experiments within six hours.
Takeaways & Limitations
The released tools, benchmarks, and Huawei deployment lessons are intended to guide parser selection, production deployment, and future research.
Abstract
from arXiv · showhide
Logs are imperative in the development and maintenance process of many software systems. They record detailed runtime information that allows developers and support engineers to monitor their systems and dissect anomalous behaviors and errors. The increasing scale and complexity of modern software systems, however, make the volume of logs explodes. In many cases, the traditional way of manual log inspection becomes impractical. Many recent studies, as well as industrial tools, resort to powerful text search and machine learning-based analytics solutions. Due to the unstructured nature of logs, a first crucial step is to parse log messages into structured data for subsequent analysis. In recent years, automated log parsing has been widely studied in both academia and industry, producing a series of log parsers by different techniques. To better understand the characteristics of these log parsers, in this paper, we present a comprehensive evaluation study on automated log parsing and further release the tools and benchmarks for easy reuse. More specifically, we evaluate 13 log parsers on a total of 16 log datasets spanning distributed systems, supercomputers, operating systems, mobile systems, server applications, and standalone software. We report the benchmarking results in terms of accuracy, robustness, and efficiency, which are of practical importance when deploying automated log parsing in production. We also share the success stories and lessons learned in an industrial application at Huawei. We believe that our work could serve as the basis and provide valuable guidance to future research and deployment of automated log parsing.
I. INTRODUCTION
Logs support software development and maintenance, but their growing volume and unstructured free-text content make manual inspection and automated analysis difficult. Automated log parsing addresses this by extracting event templates and runtime parameters, motivating a comprehensive benchmark and reusable toolkit.
- Logs record runtime information that supports usage analysis, security, anomaly detection, and diagnosing errors and crashes.
- Gigabytes of logs per hour and inherently unstructured messages make manual inspection and automated analysis difficult.
- Each message combines constant template text with variable runtime values that automated parsing must separate.
- Handcrafted parsing rules are time-consuming and error-prone, while frequent logging-code changes require continual maintenance.
- The study releases an open-source toolkit with 13 methods and evaluates them on 16 datasets for accuracy, robustness, and efficiency.
II. LOG PARSING
Log parsing structures textual messages so downstream analysis can search, group, count, and mine events. The paper motivates this role through applications including usage analysis, anomaly detection, duplicate issue identification, performance modeling, and failure diagnosis.
- Structured log events enable downstream search, filtering, grouping, counting, and sophisticated mining.
- Motivating Applications: Usage analysis uses structured events for behavior analysis, API profiling, metrics counting, and workload modeling.
- Motivating Applications: Log parsing is necessary preprocessing for machine-learning-based anomaly detection from execution logs.
- Motivating Applications: Structured event data support duplicate issue identification, performance-model construction, and automated failure diagnosis.
B. Characteristics of Log Parsers
Log parsers differ in technique, processing mode, efficiency, coverage, openness, and industrial use. These characteristics determine how suitable they are for varied log streams and practical deployments.
- Characteristics: Research parsers use strategies including frequent pattern mining, clustering, iterative partitioning, longest common subsequence, parsing trees, evolutionary algorithms, and heuristics.
- Characteristics: Offline parsers require all log data beforehand, whereas online parsers process messages one by one as streams.
- Characteristics: Efficiency matters because slow parsing can hinder low-latency tasks such as real-time anomaly detection and performance monitoring.
- Characteristics: Coverage measures whether a parser successfully structures all input messages; methods such as SLCT may fail on rare templates.
- Characteristics: Open-source availability supports reuse and improvement, while reported industrial deployment indicates practical value and reliability.
C. Techniques of Log Parsers
The study organizes its review around the techniques used by 13 log parsers.
- The paper summarizes 13 log parsers according to the techniques they use.
1) Frequent Pattern Mining:
Frequent pattern mining treats event templates as frequently occurring constant-token sets and applies this idea to automated log parsing. The cited parsers are offline methods with related procedures, while LogCluster extends SLCT and handles token-position shifts.
- Frequent Pattern Mining: Frequent pattern mining models event templates as sets of constant tokens that occur frequently in logs.
- Frequent Pattern Mining: SLCT, LFA, and LogCluster are offline frequent-pattern parsers that follow similar parsing procedures.
- Frequent Pattern Mining: LFA uses token-frequency distributions within each message to parse rare log messages.
- Frequent Pattern Mining: LogCluster extends SLCT and is robust to shifts in token positions.
3) Heuristics:
Heuristic and related parsing methods exploit log-specific structure through grouping, streaming sequence comparison, or optimization, while the toolkit standardizes access to 13 methods and production-oriented evaluation criteria.
- Heuristics: AEL, IPLoM, and Drain use heuristics tailored to unique characteristics of log messages.
- Heuristics: AEL groups messages by comparing constant-token and variable-token occurrences, while IPLoM partitions them by length, position, and mapping relations.
- Heuristics: Spell parses log streams with longest common subsequence, whereas MoLFI formulates parsing as a multiobjective optimization problem solved with evolutionary algorithms.
- Tool Implementation: The logparser toolkit unifies 13 parsing methods behind a standard interface and produces structured logs and event-template files for downstream mining.
- Evaluation Criteria: The benchmark evaluates parsers on accuracy, robustness, and efficiency, which are key qualities for production deployment.
A. Experimental Setup
The experimental setup uses the diverse LogHub collection as a benchmark, with sampled and manually labeled messages supporting reproducible evaluation of parsing accuracy and broader parser behavior.
- Dataset: LogHub contains 440 million log messages totaling 77 GB from 16 systems across multiple software domains.
- Dataset: The released datasets include production logs and logs collected from real-world systems in the authors’ laboratory, and are freely accessible for research.
- Dataset: The benchmark uses LogHub’s large and diverse datasets to assess parser accuracy, robustness, and efficiency.
- Dataset: For reproducibility, 2,000 messages are randomly sampled from each dataset and manually labeled with ground-truth event templates.
- Accuracy Metric: Parsing accuracy is the ratio of correctly parsed messages, where correctness requires matching the ground-truth message grouping.
- Experimental Controls: All parsers receive the same preprocessing, undergo more than 10 tuning runs, and are tested on a server with 32 CPUs and 62GB RAM.
B. Accuracy of Log Parsers
Accuracy is evaluated on sampled 2,000-message subsets across 16 datasets because some parsers cannot process original datasets reasonably quickly. Results vary by parser and log complexity: Drain performs best on average, while complex logs remain difficult.
- Experimental Design: 2,000-message samples are used for accuracy experiments because some parsers cannot process original datasets in reasonable time.
- Accuracy Results: Most datasets are parsed above 90% accuracy by at least one parser, while HDFS and Apache reach 100% accuracy for some parsers.
- Accuracy Results: Eight of 13 parsers achieve the best accuracy on at least two datasets.
- Accuracy Results: OpenStack, Linux, Mac, and HealthApp remain difficult to parse because of complex structures and abundant event templates.
- Accuracy Results: Drain is the most accurate parser on average and achieves high accuracy on 9 of 16 datasets; IPLoM, AEL, and Spell do so on 6 datasets each.
- Interpretation: The results suggest that effective parsers should exploit inherent log-message structure rather than directly applying standard clustering or frequent-pattern-mining algorithms.
C. Robustness of Log Parsers
The evaluation examines robustness across log types and volumes, finding substantial dataset-dependent variation and declining or fluctuating accuracy as volume increases. Drain offers the strongest average accuracy and variance profile, while efficiency also varies with parser, dataset, and log size.
- Robustness across log types: Drain achieves the highest average accuracy and the smallest variance across the 16 datasets.
- Robustness across log types: Most parsers reach maximum accuracy above 0.9, but none performs well on every log dataset.
- Robustness across log volumes: Six high-performing parsers are evaluated for robustness across different log volumes using HDFS, BGL, and Android.
- Robustness across log volumes: As log volume increases from 300 KB to 1 GB, all six parsers either lose accuracy or show obvious fluctuations.The parsers use parameters tuned on 2k-log samples, which do not fit large log data well.
- Efficiency: Parsing time increases with log size; Drain and IPLoM scale linearly and finish parsing 1GB within tens of minutes.LenMa and MoLFI cannot finish parsing 1GB of BGL or Android data within six hours.
IV. INDUSTRIAL DEPLOYMENT
Huawei deployed automated log parsing in LogKit because handcrafted rules became difficult to maintain as System X evolved. The deployment selected and optimized Drain, while identifying unresolved challenges in state identification, variable-length messages, and parameter tuning.
- Industrial motivation: System X uses logs throughout its product lifecycle for failure diagnosis, performance optimization, user profiling, and resource allocation.
- Industrial motivation: LogKit automates log search, rule-based diagnosis, and dashboard reporting, with structured log parsing as a key feature.
- Industrial motivation: Handcrafted parsing rules became unmanageable because they could not cover all log types and System X's log structures changed frequently.
- Deployment and optimization: The product team chose Drain for its superiority in accuracy, robustness, and efficiency, then optimized it for System X logs.
- Deployment and optimization: Preprocessing filters common parameters and deduplication addresses repeated constant messages, simplifying subsequent parsing.
- Potential improvements: Current parsers still struggle with state identification, variable-length messages, and automated parameter tuning.
V. RELATED WORK
Related work spans log quality, parsing, and analysis, while this paper focuses on data-driven parsing and addresses limited public evaluation resources.
- Log management research covers log quality, log parsing, and downstream log analysis.
- Log quality: Log-quality studies improve logging guidance, placement, informativeness, and runtime flexibility during software development and operation.
- Log parsing: Log parsing is categorized into rule-based, source code-based, and data-driven approaches, with this paper focusing on data-driven parsing.
- Log analysis: Log analysis applications include anomaly detection, problem diagnosis, runtime verification, and performance modeling.
- The paper implements 13 log parsing methods, evaluates them on 16 datasets, and releases the toolkit and benchmarks for reuse.