PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey on Automated Log Analysis for Reliability Engineering

Shilin He, Pinjia He, Zhuangbin Chen, Tianyi Yang, Yuxin Su, Michael R. Lyu

arXiv:2009.07237v2cs.SE

TL;DR

Automated log analysis supports reliability engineering, but the diversity of research makes the state of the art difficult for non-experts to navigate. This survey reviews 158 papers across logging, compression, parsing, mining, toolkits, and datasets, and synthesizes advances and future directions.

  • Problem

    The rapid growth and importance of software logs creates a need for effective and efficient automated analysis to support reliability engineering.

  • Method

    The survey examines 158 papers from 1997 to 2020 across logging, log compression, log parsing, log mining, open-source toolkits, and datasets.

  • Results

    The survey synthesizes the advantages and limitations of state-of-the-art approaches and identifies insights and future directions for automated log analysis.

  • Takeaways & Limitations

    The article helps outsiders enter the field and experts identify gaps in their knowledge of automated log analysis.

  • Takeaways & Limitations

    The survey excludes external attacks and intrusions because it focuses on anomalies that directly affect software reliability.

Abstract

from arXiv · show

Logs are semi-structured text generated by logging statements in software source code. In recent decades, software logs have become imperative in the reliability assurance mechanism of many software systems because they are often the only data available that record software runtime information. As modern software is evolving into a large scale, the volume of logs has increased rapidly. To enable effective and efficient usage of modern software logs in reliability engineering, a number of studies have been conducted on automated log analysis. This survey presents a detailed overview of automated log analysis research, including how to automate and assist the writing of logging statements, how to compress logs, how to parse logs into structured event templates, and how to employ logs to detect anomalies, predict failures, and facilitate diagnosis. Additionally, we survey work that releases open-source toolkits and datasets. Based on the discussion of the recent advances, we present several promising future directions toward real-world and next-generation automated log analysis.

1 INTRODUCTION

Modern software reliability depends heavily on logs, yet growing scale makes manually designed logging and rule-based analysis inefficient. This survey organizes automated log analysis research across logging, compression, parsing, mining, open-source resources, and future directions.

  • Software logs often provide the only available record of runtime information and support reliability assurance and industrial decision making.
  • Manual domain knowledge and ad-hoc rules become inefficient and ineffective as software systems grow larger and more structurally complex.
  • It surveys 158 papers from software engineering, systems, and networking venues to characterize state-of-the-art advantages, limitations, toolkits, and datasets.
  • The survey covers automated logging, runtime log compression, event-template parsing, and log mining for anomaly detection, failure prediction, and diagnosis.
  • The paper also summarizes industry practices and identifies open challenges and future directions for real-world and next-generation automated log analysis.

2 SURVEY METHODOLOGY

The survey systematically assembles and classifies automated log analysis research, while also explaining the practical scope and challenges of logging-related work. It finds sustained field growth and broad research activity, especially in log mining.

  • The authors searched major digital libraries using log-analysis keywords and expanded the collection by manually inspecting references.
  • The repository contains 158 publications spanning 1997–2020, with steady growth in annual publications since 2006.
  • The publications are classified into logging, log compression, log parsing, log mining, and empirical study according to their major contributions.
  • Around half of the research addresses log mining, which includes sub-tasks such as anomaly detection and failure diagnosis.
  • Logging: Logging research addresses where-to-log, what-to-log, and how-to-log challenges involving placement, information content, maintenance, and anti-patterns.
  • The survey groups proposed solutions into static code analysis, machine learning, and empirical study approaches.

4 LOG COMPRESSION

Log compression addresses the storage burden of continuously generated logs while preserving information needed for troubleshooting. The survey compares compression approaches by their effectiveness, speed, applicability, and handling of log-specific redundancy.

  • Challenges for Log Compression: 50 gigabytes per hour illustrates the storage burden generated by large-scale software systems running continuously.Long-term storage also supports duplicate-problem identification and mining failure patterns from historical logs.
  • Challenges for Log Compression: Reducing logging or lowering verbosity saves space but loses information, while general compressors do not exploit repetitive structured fields in semi-structured logs.These limitations motivate compression algorithms tailored to log formats.
  • Characteristics for Log Compression: A desirable compressor combines a high compression ratio with short compression and decompression time.The compression ratio is the ratio between file sizes before and after compression; higher values indicate less storage consumption.
  • Categories of Log Compression: The survey categorizes log compressors as bucket-based, dictionary-based, or statistics-based according to their principal techniques.Bucket methods partition and compress blocks in parallel; dictionary methods replace strings with references; statistics-based methods model redundancy.
  • Bucket-based Compression: Bucket-based methods exploit locality or similarity by grouping logs into blocks and compressing those blocks in parallel.LogPack illustrates layered handling of local redundancy, token repetition, and remaining redundancy.
  • Dictionary- and Statistics-based Compression: Dictionary-based methods replace repetitive strings with dictionary references, whereas statistics-based methods automatically mine compression rules from log redundancy.Statistics-based approaches can identify frequent patterns and dynamically characterize and combine log entries.

5 LOG PARSING

Log parsing converts semi-structured messages into structured events required by downstream log-mining tools. The survey examines automated parsers by their operating mode, coverage, and preprocessing requirements, while highlighting the difficulty of parsing developer-written message content.

  • Motivation: Downstream log-mining tools commonly require structured events or matrices, making automated parsing a crucial analysis step.Raw logs are semi-structured messages that must be transformed before tasks such as anomaly detection.
  • Parsing Objective: Parsing separates fixed constants describing system events from variable runtime values in free-form log message content.The message header is comparatively easy to extract because it is determined by the logging framework.
  • Parsing Output: A parsed log message contains an event template formed from constants and key parameters extracted from variables.This structured representation supplies the event-level information used by subsequent analysis.
  • Practical Limitations: Industrial log-management solutions often support common log types, but general messages frequently require developer-provided regular expressions and ad-hoc scripts.These scripts group messages expected to share an event template.
  • Survey Scope: The survey compares 15 automated log parsing approaches using mode, coverage, and preprocessing as principal characteristics.Its methodological comparison differs from prior evaluation work that provided benchmarks and open-source implementations.
  • Mode: Offline parsers batch-process all available messages and must be rerun after software updates, whereas online parsers process messages as a stream.Online parsing can operate alongside downstream log-mining tasks.

Mode.

Coverage measures whether a parser matches all input log messages with event templates, independently of whether those templates are correct.

  • Coverage: Coverage is the capability to match all input log messages with event templates, and it is orthogonal to template correctness.“Partial” coverage means some messages receive no event template.

Coverage.

Preprocessing uses domain knowledge to remove variables or replace them with constants before parsing.

  • Preprocessing: Preprocessing removes or normalizes variables such as IP addresses before parsing, typically using manually constructed regular expressions.The step therefore introduces manual effort based on domain knowledge.

Preprocessing.

Offline log parsers use token patterns, clustering, heuristics, optimization, or hierarchical partitioning to generate event templates from log messages.

  • Preprocessing.: SLCT counts frequent tokens in two passes and uses associated frequent words as constants for event-template generation.Messages lacking the associated pattern are placed in an outlier cluster without matched templates.
  • Preprocessing.: LFA adopts a strategy similar to SLCT but covers all log messages.
  • Preprocessing.: LogCluster uses clustering to handle variable-length parameters between fixed words in log messages.For example, different application names can share the template “Download <*> and install.”
  • Preprocessing.: LKE, LogSig, and LogMine apply hierarchical clustering, word-pair representations, and agglomerative clustering, respectively.
  • Preprocessing.: AEL uses heuristic rules, while MoLFI formulates parsing as multi-objective optimization with an evolutionary algorithm.MoLFI requires little parameter tuning because its four parameters have effective default values.
  • Preprocessing.: IPLoM hierarchically partitions messages by length, token position, and token mapping.

Iterative Partitioning.

Large-scale and online parsing approaches emphasize distributed execution, tree-based grouping, specialized data structures, automated tuning, and frequent n-gram statistics.

  • Iterative Partitioning.: POP uses distributed computing and parses 200 million HDFS log messages in 7 minutes, while many parsers failed to terminate reasonably.
  • Iterative Partitioning.: SHISO is the first online log parsing approach and uses a manually configured tree whose nodes correspond to log groups and event templates.Its parsing process traverses the tree by comparing messages with event templates.
  • Iterative Partitioning.: Spell accelerates online parsing with a prefix tree, inverted index, and parallel implementation.
  • Iterative Partitioning.: Drain maintains log groups through tree leaf nodes and uses heuristic rules in internal nodes; its extension supports online merging and automated parameter tuning.
  • Iterative Partitioning.: Logram treats frequent n-grams as likely constants and assumes developers have log messages available to construct its dictionary.

6 LOG MINING

Log mining automates the extraction of patterns from large log volumes for monitoring, administration, troubleshooting, and reliability tasks. Its workflow partitions logs, extracts features, trains models, and deploys them online.

  • 6 LOG MINING: Log mining applies statistics, data mining, and machine learning to discover meaningful patterns and trends in large log datasets.The extracted knowledge supports monitoring, administration, and troubleshooting.
  • 6.1 Challenges of Log Mining: Keyword search and manually defined rules become ineffective as modern logs increase in volume, variety, and velocity.Manual inspection of massive logs makes failure-relevant messages difficult to identify.
  • 6.1 Challenges of Log Mining: Heterogeneous logs from many components and diverse software or hardware failure sources make troubleshooting difficult for a single engineer.
  • 6.2 General Workflow of Log Mining: The general log-mining workflow consists of log partition, feature extraction, model training, and online deployment.
  • 6.2.1 Log Partition: Interleaved logs from microservices and multithreaded modules should be partitioned into groups representing individual system tasks before feature extraction.
  • 6.2.1 Log Partition: Timestamp-based partitioning commonly uses fixed or sliding windows, while log identifiers distinguish logs from different task executions.Identifiers may be non-unique across entities because they are not always propagated or synchronized across services, and threads or processes may multiplex requests.
  • 6.2.2 Feature Extraction: Log-mining features are numerical or graphical: numerical vectors encode log statistics, while directed graphs represent dependencies, co-occurrence, and execution behavior.Numerical features include event sequences, event-count vectors, and parameter-value vectors; graphical features support monitoring and diagnosis.
  • 6.4 Failure Prediction: In heterogeneous systems, critical-failure prediction models relationships among multiple components using failure signals collected from logs.

Specification Mining.

Specification mining extracts execution constraints from program traces, while log-based testing uses generated or recorded logs to stimulate workflows and improve testing coverage.

  • Specification Mining.: Specification mining extracts program specifications such as invariants from execution traces.
  • Specification Mining.: The extracted specifications constrain execution ordering and program values and support system comprehension, verification, and evolution.
  • Specification Mining.: Log-based software testing stimulates software workflows with generated log data and commonly uses state-machine formalisms.Existing approaches use log files to improve testing coverage or completeness.

7 OPEN-SOURCE TOOLKITS AND DATASETS

The survey reviews open-source toolkits and public datasets that support automated log analysis across software systems and research settings.

  • Open-Source Toolkits: Eight open-source toolkits support features including log collection, searching, routing, parsing, visualization, alerting, and automated analysis.The listed tools are GrayLog, GoAccess, Fluentd, Logstash, Logalyze, Prometheus, Syslog-ng, and LogPAI.
  • Open-Source Toolkits: Most existing tools focus on log-analysis phases before mining, such as parsing, and on resulting outcomes such as visualization.
  • Public Datasets: Loghub maintains a large, freely accessible log collection for academic research, alongside production and laboratory datasets.
  • Public Datasets: Public datasets include failure-injected OpenStack logs, component failure logs from production systems, security data, and Apache web-search logs.

8 CONCLUSION

The survey synthesizes automated log analysis research across logging, compression, parsing, and mining, while reviewing available tools and datasets. It identifies future directions for agile and distributed development and for next-generation log analysis frameworks.

  • The survey examines four main steps in automated log analysis: logging, log compression, log parsing, and log mining.
  • It introduces available open-source toolkits and datasets to support understanding and use of the field.
  • The survey proposes future directions toward automated log analysis that better fits modern agile and distributed development.
  • It discusses a next-generation log analysis framework as a future direction.
Loading 2009.07237v2…