PaperlayerView exact revision on arXiv

Source-linked AI summary

Loghub: A Large Collection of System Log Datasets for AI-driven Log Analytics

Jieming Zhu, Shilin He, Pinjia He, Jinyang Liu, Michael R. Lyu

arXiv:2008.06448v3cs.SE

TL;DR

AI-driven log analytics lacks public datasets and open benchmarks despite the growing volume and importance of system logs. This paper releases Loghub, a diverse collection of 19 datasets, and benchmarks representative analytics tasks. The benchmarks report strong but dataset-dependent parser performance and identify unresolved challenges across parsing, compression, and anomaly detection.

  • Problem

    Public log datasets and standard benchmarks are limited, hindering reproducible evaluation of AI-driven log-analytics approaches across diverse systems.

  • Method

    The paper collects and organizes 19 real-world log datasets, presents usage scenarios, and benchmarks log parsing, compression, and anomaly detection.

  • Results

    Over 90% accuracy is achieved by at least one log parser on most datasets, while complex datasets such as OpenStack, Linux, Mac, and HealthApp remain challenging.

  • Takeaways & Limitations

    Loghub provides an open benchmark for AI-driven log analytics and supports research across diverse datasets and tasks.

Abstract

from arXiv · show

Logs have been widely adopted in software system development and maintenance because of the rich runtime information they record. In recent years, the increase of software size and complexity leads to the rapid growth of the volume of logs. To handle these large volumes of logs efficiently and effectively, a line of research focuses on developing intelligent and automated log analysis techniques. However, only a few of these techniques have reached successful deployments in industry due to the lack of public log datasets and open benchmarking upon them. To fill this significant gap and facilitate more research on AI-driven log analytics, we have collected and released loghub, a large collection of system log datasets. In particular, loghub provides 19 real-world log datasets collected from a wide range of software systems, including distributed systems, supercomputers, operating systems, mobile systems, server applications, and standalone software. In this paper, we summarize the statistics of these datasets, introduce some practical usage scenarios of the loghub datasets, and present our benchmarking results on loghub to benefit the researchers and practitioners in this field. Up to the time of this paper writing, the loghub datasets have been downloaded for roughly 90,000 times in total by hundreds of organizations from both industry and academia. The loghub datasets are available at https://github.com/logpai/loghub.

I. INTRODUCTION

Logs provide rich runtime information for software development and maintenance, but their rapid growth makes manual analysis costly. Loghub addresses limited public data and benchmarking by collecting diverse system-log datasets for AI-driven analytics.

  • Logs record runtime information that developers and operators use to track system behavior and perform post-mortem analysis.
  • AI techniques target automated tasks including anomaly detection, duplicate issue identification, usage statistics analysis, and program verification.
  • Rapidly growing log volumes, reaching 50 GB/hour in one example, make manual analysis labor-intensive and time-consuming.
  • Researchers often use private log data, so methods effective on one log type may become ineffective on another.
  • Loghub provides 19 datasets from diverse systems to help bridge the gap between log-analytics research and practice.

A Sample Code Snippet

Loghub is a freely available collection of diverse system logs organized for research and practice. It supports multiple analytics tasks through labeled and unlabeled datasets, usage scenarios, and benchmarks.

  • Loghub contains 19 datasets generated by distributed systems, supercomputers, operating systems, mobile systems, server applications, and standalone software.
  • Six datasets are labeled for tasks such as anomaly detection and duplicate issue identification, while other datasets support parsing, compression, and unsupervised anomaly detection.
  • Since release, Loghub datasets have been downloaded more than 90,000 times by over 450 organizations from industry and academia.
  • The paper introduces practical usage scenarios and benchmarks three typical log-analysis tasks using Loghub.
  • The collection totals over 77 GB of logs, including production data from previous studies and logs collected from real systems in the authors’ lab.

A. Distributed Systems

Loghub’s distributed-systems datasets cover HDFS, Hadoop, Spark, ZooKeeper, and OpenStack. They vary in collection environments, aggregation levels, labels, workloads, and research uses.

  • HDFS: HDFS contributes three datasets: HDFS-v1 is labeled by block-level traces, HDFS-v2 contains over 16 GB of unmodified unlabeled logs, and HDFS-v3 comes from a real IaaS environment.
  • HDFS: HDFS-v1 includes normal or abnormal labels and specific anomaly types, supporting anomaly detection and duplicate issue identification.
  • Hadoop: Hadoop logs come from a five-machine cluster running WordCount and PageRank with normal executions and injected deployment failures.
  • Spark: Spark logs were aggregated at machine level from a 32-machine lab system and contain over 2 GB of unmodified, unlabeled data.
  • ZooKeeper: The ZooKeeper dataset covers 26.7 days of logs aggregated from a 32-machine lab environment.
  • OpenStack: OpenStack provides normal logs and failure-injected abnormal cases for anomaly-detection research.

B. Supercomputers

The collection includes datasets from supercomputers and other systems, with some labeled for alert detection and prediction research. It also includes operating-system, mobile, and server logs.

  • B. Supercomputers: BGL and Thunderbird contain alert and non-alert messages identified by alert category tags.Their labels support alert detection and prediction research.
  • B. Supercomputers: HPC logs were collected from System 20 of the Los Alamos National Laboratories high-performance computing cluster.The cluster has 49 nodes, 6,152 cores, and 128 GB memory per node.
  • C. Operating Systems: The Windows dataset exceeds 27 GB and spans 226.7 days, while Linux logs were collected from a server over 263.9 days.Windows logs came from CBS; Linux logs came from /var/log/messages.
  • C. Operating Systems: Mac logs were collected from a personal MacBook after seven days of use and record user activities.
  • D. Mobile Applications: Android-v1 is a sampled small log file from Android-v2, whose logs cover two issue types with more than 10 duplicate issue logs per type.The paper states that Android’s complex multithreading makes abnormal log points difficult to pinpoint.
  • D. Mobile Applications: HealthApp logs were collected from an Android smartphone after more than 10 days of use, while Apache provides an error log for anomaly detection and diagnosis.

F. Standalone Software

Loghub includes standalone software logs and is used across diverse research and education scenarios. The collection’s reported adoption spans companies, universities, and startup log-analysis firms.

  • F. Standalone Software: Proxifier logs were collected from a desktop computer in the authors’ lab.Proxifier enables network applications without proxy support to operate through SOCKS or HTTPS proxies and chains.
  • A. Overview: The datasets have attracted large companies, startup log-analysis firms, and universities.
  • A. Overview: Loghub’s usage scenarios include anomaly detection, log analysis, security, log parsing, and education among 23 categorized purposes.The categorization is rough because users provided limited information.

B. Log Parsing

Log parsing converts unstructured log messages into structured system events needed by many AI-based analyses. Loghub supplies large and diverse datasets for evaluating parsing and related log-analysis approaches.

  • B. Log Parsing: Log parsing transforms unstructured log messages into structured system events for AI-driven log analytics.AI-based approaches often require event lists with IDs or matrices as input.
  • B. Log Parsing: Data-driven parsers automatically label unstructured log messages with corresponding system event IDs, and parsing is typically modeled as clustering.
  • B. Log Parsing: 19 Loghub datasets from 6 system categories contain over 77 GB of logs for evaluating parsing accuracy and efficiency.
  • Log Compression: Log compression addresses long-term storage costs because general compression approaches do not account for log-message structure.
  • Log Compression: Log compression can be modeled as frequent pattern mining that detects repetitive text to achieve efficient, lossless compression with a high compression rate.
  • Log Compression: All Loghub datasets can support evaluating log compression approaches because the task requires large volumes of logs from diverse systems.
  • Anomaly Detection: Log-based anomaly detection targets abnormal system behavior in large, complex systems where downtime can cause enormous revenue loss.
  • Anomaly Detection: Anomaly detection uses structured events or matrices to classify instances as abnormal or normal through supervised or unsupervised approaches.

E. Duplicate Issues Identification

Duplicate issue identification clusters operational issues with similar log-message patterns so developers can handle user-reported problems more efficiently. Loghub provides labeled datasets and benchmarks related log-analysis methods.

  • E. Duplicate Issues Identification: Operational issue handling requires inspecting runtime logs to understand system operations, which is time-consuming.The paper illustrates this with a user-reported Amazon EC2 node-slowness issue.
  • E. Duplicate Issues Identification: Duplicate issue identification clusters issues whose log messages have similar patterns, including occurrence frequency and order.
  • E. Duplicate Issues Identification: Loghub provides 3 labeled datasets—HDFS-v1, Hadoop, and Android-v2—with issue categories for evaluating duplicate issue identification.
  • IV. BENCHMARKING ON LOGHUB DATASETS: The paper benchmarks typical log-analysis tasks including log parsing, log compression, and log-based anomaly detection.The benchmarking is used to derive unresolved questions and challenges.
  • Benchmarking: The paper evaluates 13 log parsing algorithms categorized as frequent pattern-based, clustering-based, heuristics-based, and other methods.

1) Existing Log Parsing Algorithms:

Log parsing methods use frequent patterns, clustering, heuristics, or other strategies to group unstructured log messages into event templates. Loghub benchmarks these approaches across datasets, revealing strong but uneven performance and persistent challenges on complex logs.

  • Method categories: Frequent pattern-based methods identify recurring tokens or token-position pairs, then cluster messages and extract templates.SLCT, LFA, and LogCluster require the complete log set before parsing.
  • Method categories: Clustering-based methods group log messages with core algorithms such as hierarchical clustering.
  • Method categories: Heuristics-based parsers use features such as token frequency, message length, token location, mapping connections, or fixed-depth trees.Examples include AEL, IPLoM, and Drain.
  • Evaluation: Parsing accuracy is the fraction of correctly parsed logs among all logs, requiring each message's template to match the ground-truth cluster.A sequence parsed as [E1, E4, E5] instead of [E1, E2, E2] has PA 1/3.
  • Benchmark results: Over 90% accuracy is achieved by at least one parser on most datasets, and 8 out of 13 parsers are best on at least two datasets.HDFS and Apache can reach 100% accuracy because their event templates are simpler.
  • Challenges: No single parser handles every dataset effectively, while complex logs such as Mac, with 341 templates, remain difficult.Drain has the highest average accuracy and high precision on 9 out of 16 datasets.

3) Remaining Questions and Challenges:

The compression benchmark examines how existing tools perform across Loghub datasets while highlighting unresolved trade-offs between compression effectiveness, template diversity, searchability, and resource usage.

  • Challenges: Log-specific compression performance varies substantially across datasets, and sparse log messages can decrease parsing accuracy.
  • Benchmark setup: Loghub is used in a case study benchmarking existing log compression algorithms.
  • Benchmark setup: Six compression tools are evaluated, including log-specific tools and general tools such as gzip, bzip2, and lzma.Logzip is tested with three compression kernels.
  • Metric: Compression ratio is used to evaluate effectiveness, with smaller compressed files producing higher CR values.

2) Benchmarking on Loghub:

Loghub supports benchmarking across log-analysis tasks, including compression and anomaly detection, using diverse datasets and task-specific evaluation metrics. The supplied passages emphasize compression results and anomaly-detection evaluation design.

  • Compression evaluation: A smaller compressed file yields a higher compression ratio, indicating a more effective compression tool.
  • Compression evaluation: Five large-scale Loghub datasets are used to benchmark compression tools in terms of CR.The results are reported in Table IV.
  • Compression results: Logzip outperforms the other evaluated methods on all five datasets, averaging 4.56x CR and reaching a maximum of 15.1x over gzip.
  • Anomaly detection: Anomaly detection is evaluated with precision, recall, and F-measure on labeled data.The approaches include supervised and unsupervised techniques such as logistic regression, Isolation Forest, PCA, invariant mining, LOF, and clustering.

2) Benchmarking on Loghub:

The anomaly-detection benchmark uses a structured HDFS representation and compares supervised with unsupervised approaches. Supervised methods perform better in the reported experiment, while additional labeled datasets enable broader evaluation.

  • Benchmark setup: The HDFS input is a block-ID-by-event count matrix, with rows representing block operations and columns representing event frequencies.
  • Supervised approaches: Decision Tree achieves recall 0.99, F-measure 0.99, and the second-highest precision of 0.99.
  • Comparison: Supervised approaches achieve better results than unsupervised approaches in the reported benchmark because they are trained on labeled data.
  • Unsupervised approaches: Among unsupervised approaches, Invariants Mining has the best accuracy, while Clustering reaches precision 1.00 but recall 0.72.
  • Unsupervised approaches: One-Class SVM obtains precision 0.99 but recall 0.22, producing a low F-measure because it conservatively reports anomalies.
  • Dataset scope: Loghub provides four additional labeled datasets for evaluating anomaly-detection methods and selecting suitable datasets in practice.

3) Remaining Questions and Challenges:

Log analytics still faces challenges spanning data availability, operational usefulness, scalability, and changing software behavior. Loghub addresses the research–practice gap by providing shared datasets, usage scenarios, and benchmarks for common analysis tasks.

  • Remaining Questions and Challenges: Existing anomaly detection research still faces challenges including limited diagnosis support, weak visualization, runtime efficiency, and concept drift.Current approaches often report only anomaly status, focus on log sequences, prioritize accuracy, and struggle as log data evolves.
  • Logging Practice: Rigorous guidance and specifications for developer logging behavior remain lacking in practice.Related research studies logging practices and AI-based decisions about where and what to log.
  • Log Compression: Log compression remains a specialized research area because software logs contain inherent repetitive structure.Existing methods exploit domain knowledge and repeated content to improve compression rates.
  • Log Parsing: Log parsing is crucial because many automated log analysis techniques require structured data as input.Parsing transforms unstructured log messages into structured system events, using approaches such as frequent-pattern mining and clustering.
  • Log Analysis: Log analysis covers tasks including anomaly detection, duplicate issue identification, incident diagnosis, usage statistics analysis, and program verification.Most of these tasks design or adopt AI algorithms to support effective and efficient system maintenance.
  • Loghub: Loghub provides 19 datasets totaling over 77 GB, with usage scenarios and benchmarks for parsing, compression, and anomaly detection.The authors position it as an open benchmarking system and identify future expansion and a benchmarking leaderboard as directions.
Loading 2008.06448v3…