PaperlayerView exact revision on arXiv

Source-linked AI summary

A Large-Scale Evaluation for Log Parsing Techniques: How Far Are We?

Zhihan Jiang, Jinyang Liu, Junjie Huang, Yichen Li, Yintong Huo, Jiazhen Gu, Zhuangbin Chen, Jieming Zhu, Michael R. Lyu

arXiv:2308.10828v2cs.SE

TL;DR

Loghub-2k’s limited scale and message-level metrics leave gaps in evaluating parsers on production-like, imbalanced logs. The paper constructs Loghub-2.0 and re-evaluates 15 parsers with template-level and granular assessments. Results show poorer performance on the larger benchmark, especially for rare and parameter-intensive logs, and efficiency constraints for many parsers.

  • Problem

    Loghub-2k annotates only 2,000 lines per dataset, while message-level metrics can favor frequent templates and whole-dataset scores omit fine-grained performance on important log types.

    Source: §1 · p. 2, §2.2 · p. 3

  • Method

    The authors create 14 manually annotated Loghub-2.0 datasets averaging 3.6 million lines and evaluate 15 parsers using template-level FGA and granular log-category assessments.

    Source: §1 · p. 2, §4.2 · p. 6

  • Results

    Parsers perform less satisfactorily on Loghub-2.0, particularly on infrequent and parameter-intensive logs; Drain has the highest average GA and FGA, while most parsers fall short on large-scale efficiency.

    Source: §5.4 · p. 10

  • Takeaways & Limitations

    The evaluation indicates that parser assessment should account for template characteristics and large-scale processing, not only overall dataset performance.

    Source: §1 · p. 2, §7 · p. 11

    Potential annotation errors remain because Loghub-2.0 is constructed without source code, and the parser selection is limited by open-source availability.

    Source: §6.2 · p. 11

Abstract

from arXiv · show

Log data have facilitated various tasks of software development and maintenance, such as testing, debugging and diagnosing. Due to the unstructured nature of logs, log parsing is typically required to transform log messages into structured data for automated log analysis. Given the abundance of log parsers that employ various techniques, evaluating these tools to comprehend their characteristics and performance becomes imperative. Loghub serves as a commonly used dataset for benchmarking log parsers, but it suffers from limited scale and representativeness, posing significant challenges for studies to comprehensively evaluate existing log parsers or develop new methods. This limitation is particularly pronounced when assessing these log parsers for production use. To address these limitations, we provide a new collection of annotated log datasets, denoted Loghub-2.0, which can better reflect the characteristics of log data in real-world software systems. Loghub-2.0 comprises 14 datasets with an average of 3.6 million log lines in each dataset. Based on Loghub-2.0, we conduct a thorough re-evaluation of 15 state-of-the-art log parsers in a more rigorous and practical setting. Particularly, we introduce a new evaluation metric to mitigate the sensitivity of existing metrics to imbalanced data distributions. We are also the first to investigate the granular performance of log parsers on logs that represent rare system events, offering in-depth details for software diagnosis. Accurately parsing such logs is essential, yet it remains a challenge. We believe this work could shed light on the evaluation and design of log parsers in practical settings, thereby facilitating their deployment in production systems.

1 INTRODUCTION

Loghub-2k and message-level evaluations leave questions about production-scale, imbalanced logs and parser performance on rare and parameter-intensive templates. The paper introduces Loghub-2.0 and a broader benchmark, then re-evaluates 15 parsers.

1 INTRODUCTION

  • Loghub-2k annotates only 2,000 randomly sampled lines per dataset, limiting evidence about production-scale log characteristics.The authors identify frequency and parameter count as especially relevant characteristics.Source: §1 · p. 2
  • Message-level metrics can be dominated by frequent templates, while whole-dataset results obscure performance on infrequent and parameter-intensive templates.These log types can include severe events and runtime-status details relevant to downstream analysis.Source: §1 · p. 2
  • The paper introduces Loghub-2.0, 14 manually annotated datasets averaging 3.6 million lines, and a benchmark protocol with template-level FGA and granular evaluations.The study re-evaluates 15 state-of-the-art parsers.Source: §1 · p. 2
  • All parsers degrade on Loghub-2.0 versus Loghub-2k; high overall scores do not ensure effective parsing of infrequent or parameter-intensive logs, and 9 of 15 parsers time out on some datasets.The authors also report greater performance variance on Loghub-2.0.Source: §1 · p. 2

2 BACKGROUND AND MOTIVATION

The section reviews four families of log parsers and motivates a more practical benchmark. It identifies limits in dataset scale and annotation, metric coverage, and whole-dataset-only evaluation.

2.1 Existing Log Parsing Techniques

  • Log parsers span frequency-based, similarity-based, heuristics-based, and semantic-based methods, using distinct strategies to distinguish templates and parameters.Semantic-based approaches use neural models to classify log tokens.Source: §2.1 · p. 3

2 BACKGROUND AND MOTIVATION

  • Loghub-2k, introduced as a benchmark of 13 parsers, samples and manually labels 2,000 messages per dataset.Subsequent approaches also evaluated on this widely used collection.Source: §2.2 · p. 3

2.2 Motivation

  • Loghub-2k’s 2,000 lines per dataset and potential annotation inconsistencies limit its representation of large, diverse production logs.The section also notes reported template-label errors in the dataset.Source: §2.2 · p. 3
  • Message-level metrics can favor frequent templates, allowing routine logs to dominate scores and mask errors on infrequent templates.Production log distributions may be highly imbalanced.Source: §2.2 · p. 3
  • Whole-dataset results do not reveal parser performance on infrequent event templates or parameter-intensive templates that carry system-status details.The section identifies both types as important to production maintenance.Source: §2.2 · p. 3
  • The authors propose a larger, more diverse dataset collection and a more comprehensive benchmarking protocol to address these limitations.Source: §2.2 · p. 3

3 DATASET CONSTRUCTION

Loghub-2.0 is built from 14 annotated datasets using a multi-stage process that combines log grouping, template annotation, matching, and refinement. The resulting collection substantially increases the annotated scale and template count over Loghub-2k.

3.1 Overview

  • The annotation framework uses preprocessing, log grouping, template annotation, log-template matching, and template refinement to label large datasets efficiently.Grouping and template matching reduce the need for manual line-by-line annotation.Source: §3 · p. 3, §3.1 · p. 4

3.2 Preprocessing

  • Logs are cleaned by extracting fields, removing content without alphabetic characters, and temporarily deduplicating identical lines.Source: §3.2 · p. 4

3.3 Log Grouping

  • Grouping partitions messages by logging level and component, then by shared top-K frequent tokens, with K set from one to three per dataset.Stop words are excluded from the frequent-token grouping.Source: §3.3 · p. 4

3.4 Template Annotation

  • Annotators independently label templates within lexicographically sorted groups, replacing parameter tokens with <*> and merging duplicate templates across groups.The process follows shared parameter categories and template-formatting heuristics.Source: §3.4 · p. 5

3.5 Log-Template Matching

  • Templates are converted to regular expressions for message matching; when multiple templates match, the one with longer static parts takes priority.Unmatched messages trigger template review and another matching pass.Source: §3.5 · p. 5

3.6 Template Refinement

  • Annotators’ template labels had an average consistency score of 0.926, and consensus labels became the final annotations.Source: §3.6 · p. 5

3.7 Annotation Results

  • Loghub-2.0 averages 3,601,187 annotated messages per dataset, 1,875 times Loghub-2k’s 2,000, and averages 249.1 templates versus 81.9.The average template count increased by 204.2%.Source: §3.7 · p. 5

4 STUDY DESIGN

The study compares Loghub-2.0 with Loghub-2k, re-evaluates parsers across three research questions, and uses message- and template-level metrics to assess them. Its design also specifies parser selection and experimental settings.

4.1 Research Question

  • The benchmark asks how Loghub-2.0 differs from Loghub-2k, how parser performance changes between them, and how parsers perform on logs with varying characteristics.The third question targets diverse template frequencies and parameter counts.Source: §4.1 · p. 6

4.2 Evaluation Metrics

  • Message-level Group Accuracy (GA) and Parsing Accuracy (PA) weight results by message counts, while template-level FGA and FTA give templates equal consideration.FGA is the paper’s template-level version of GA; FTA combines template-accuracy precision and recall.Source: §4.2 · p. 6, §4.2.2 · p. 6

4.2.1 Message-Level Metrics.

  • GA measures correct grouping of messages against ground-truth groups, while PA requires correctly identifying every template and parameter token in a message.PA is relevant to downstream analyses that use parameter values.Source: §4.2.1 · p. 6

4.2.2 Template-Level Metrics.

  • Message-level metrics can mask errors on rare templates: when 95% of messages belong to 1% of templates, a parser can score 0.95 despite errors on the other 99%.FGA evaluates correctly grouped templates using the harmonic mean of precision and recall; FTA additionally requires exact template-token identification.Source: §4.2.2 · p. 6

4.3 Evaluation Setup

  • The evaluation selects 15 parsers, including statistical and semantic-based methods, and excludes LKE because its pairwise-distance computation is impractical at large scale.The study reuses prior implementations where available and implements UniParser from its paper.Source: §4.3 · p. 7
  • The experiments use consistent preprocessing and tuned settings; randomized parsers are run five times and reported by median to reduce randomness-related bias.Experiments run on a server with a Xeon Gold 6226R CPU, 256GB RAM, and GTX3090 GPU.Source: §4.3 · p. 7

5.1 RQ1: Differences between Loghub-2.0 and Loghub-2k

Loghub-2.0 differs from Loghub-2k in template-frequency and parameter-count distributions. Its templates span broader frequency ranges, show greater imbalance, and include more parameters.

5.1 RQ1: Differences between Loghub-2.0 and Loghub-2k

  • In Spark, template frequencies in Loghub-2.0 range from 1 to over 10^6, compared with 1 to around 10^3 in Loghub-2k.Only 10% of Spark templates in Loghub-2.0 exceed frequency 10^4, yet those templates account for most logs.Source: §5.1 · p. 7
  • Loghub-2.0 has more imbalanced template frequencies and higher parameter counts on average than Loghub-2k.The comparison is illustrated for Spark, Linux, and OpenSSH; the full set of 14 dataset distributions is available in the repository.Source: Figure 2 · p. 7, §5.1 · p. 7
  • Spark templates reach 24 parameters in Loghub-2.0, compared with a maximum of 3 in Loghub-2k.The passage reports a similar pattern for Linux and OpenSSH.Source: §5.1 · p. 7

5.2 RQ2: Performance differences of log parsers on Loghub-2.0 and Loghub-2k

Across parsers, results on Loghub-2.0 differ from Loghub-2k: template-level scores drop and results vary more across datasets. Most parsers also fail to finish all datasets within the 12-hour limit.

5.2.1 Effectiveness.

  • Message-level GA and PA usually score higher than template-level FGA and FTA, with larger differences on the more imbalanced Loghub-2.0.For AEL, the average GA–FGA discrepancy is about 0.1 on Loghub-2k and roughly 0.3 on Loghub-2.0.Source: §5.2.1 · p. 8
  • Semantic-based parsers have higher PA and FTA but lower GA and FGA than other parsers, and their performance may decline on larger, more diverse datasets when training samples are limited.The paper attributes lower grouping scores to neglect of global information such as statistical frequency.Source: §5.2.1 · p. 8

5.2.2 Efficiency.

  • 9 out of 15 parsers cannot finish all 14 Loghub-2.0 datasets within 12 hours.Semantic-based parsers typically demand more computational resources than statistical parsers; LogPPT’s CPU time is considerably higher than efficient statistical parsers such as Drain.Source: §5.2.2 · pp. 8–9

5.2.1 Effectiveness.

  • On Loghub-2.0, parsers show performance drops and increased metric variance compared with Loghub-2k.Drain’s average FGA falls from 0.75 to approximately 0.55, while LogPPT’s average FTA falls from roughly 0.64 to 0.5.Source: §5.2.1 · p. 8, Figure 3 · p. 9

5.3 RQ3: Performances of log parsers on logs with different characteristics

The study evaluates parsers on logs with different template frequencies and parameter counts, finding distinct weaknesses across these groups. Parameter-intensive logs remain especially difficult, while frequency results differ by metric.

5.3 RQ3: Performances of log parsers on logs with different characteristics

  • The study parses full datasets before evaluating subsets, keeping parser inputs consistent with RQ2; it reports five representative parsers because of space limits.For frequency, it examines the top and bottom 5%, 10%, and 20% of template frequencies; Figure 4 shows k=10.Source: §5.3 · p. 9, §5.3.1 · p. 9

5.3.1 Performance with different template frequencies.

  • On frequent templates, all parsers have lower GA and FGA than on infrequent templates; LogPPT’s scores approach 0.95 for infrequent templates but fall below 0.5 for frequent ones.The authors attribute the grouping difficulty to having more messages to group for frequent templates.Source: §5.3.1 · p. 9, Figure 4 · p. 10
  • All parsers have lower average PA and FTA on infrequent templates, where less evidence is available to guide parsing.For semantic parsers such as LogPPT, training-log sampling also makes infrequent templates less likely to be selected.Source: §5.3.1 · p. 9

5.3.2 Performance with different parameter counts.

  • For templates with five or more parameters, all parsers perform poorly; LogPPT’s average FGA is 0.16, while other methods’ average FGA does not surpass 0.03.Drain exceeds 0.95 on all four metrics for logs without parameters, above its overall performance.Source: §5.3.2 · pp. 9–10, Figure 5 · p. 10

5.4 Summary of all research questions

Across the research questions, Loghub-2.0 presents greater scale and complexity than Loghub-2k, and existing parsers remain unsatisfactory on it. Parser strengths differ by task, while most also fall short of large-scale efficiency demands.

5.4 Summary of all research questions

  • Drain has the highest average GA and FGA, while semantic methods such as UniParser and LogPPT better distinguish constant from dynamic tokens but group messages less effectively.Token-classification errors can lead to incomplete groups.Source: §5.4 · p. 10
  • Despite encouraging Loghub-2k results, parser performance on Loghub-2.0 remains unsatisfactory, particularly for infrequent and parameter-intensive logs.Source: §5.4 · p. 10
  • The majority of evaluated log parsers do not meet the efficiency demands of large-scale application scenarios.Source: §5.4 · p. 10

6 DISCUSSION

The discussion recommends evaluating both message- and template-level performance, examining logs with different characteristics, and emphasizing efficiency. It also identifies annotation errors and limited parser selection as validity threats.

6.1 Implications

  • Message-level metrics can be dominated by frequent templates, whereas template-level metrics resist frequency imbalance; the authors recommend considering both and prioritizing by application needs.Source: §6.1 · p. 10
  • High overall scores can coexist with weak performance on infrequent and parameter-intensive templates, so evaluations should examine those groups and parser efficiency.The authors also suggest combining semantic and statistical information to improve parsing and grouping capabilities.Source: §6.1 · pp. 10–11

6.2 Threats to Validity

  • Potential annotation errors remain a threat because Loghub-2.0 was annotated without source code, despite a stringent framework and a five-member experienced team.Source: §6.2 · p. 11
  • Parser selection is limited because some tools are not open source; the selected set covers state-of-the-art parsers and existing technology categories.The authors used established or original code where available, replicated other parsers, and tuned parameters to reduce implementation and setting bias.Source: §6.2 · p. 11

7 CONCLUSION

The paper presents a large-scale log parsing evaluation using a new annotation framework, dataset collection, and benchmarking protocol. Its re-evaluation identifies limitations in existing parsers and benchmarks.

7 CONCLUSION

  • The authors annotate a new collection of log datasets intended to reflect the scale and distribution of real-world log data.Source: §7 · p. 11
  • The benchmarking protocol combines a template-level metric with evaluations across logs having varying characteristics for a more comprehensive analysis.Source: §7 · p. 11
  • Re-evaluating selected parsers with Loghub-2.0 uncovers limitations of existing log parsers and benchmarks.Source: §7 · p. 11

Related papers

Loading 2308.10828v2…