PaperlayerView exact revision on arXiv

Source-linked AI summary

LLMParser: An Exploratory Study on Using Large Language Models for Log Parsing

Zeyang Ma, An Ran Chen, Dong Jae Kim, Tse-Hsun Chen, Shaowei Wang

arXiv:2404.18001v1cs.SEcs.AI

TL;DR

Log parsing is an important first step in log analysis, yet traditional parsers can struggle to identify parameters in diverse logs. This paper evaluates few-shot LLMParser across open-source systems and finds higher parsing accuracy than state-of-the-art parsers, while model size, training shots, and log pretraining do not consistently improve results.

  • Problem

    Traditional log parsers can fail to identify parameters in diverse logs, although parsing accuracy matters because parsing is an early step in downstream log analysis.

    Source: §1 · pp. 1–2

  • Method

    LLMParser fine-tunes generative LLMs with few-shot log examples to translate raw logs into templates, evaluating four models on logs from 16 open-source systems.

    Source: §1 · p. 2, §3.1 · p. 4

  • Results

    LLMParsers achieve average PA 0.96, higher than the state-of-the-art parsers’ highest PA of 0.92.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Shot size, model size, and cross-system log pretraining do not consistently improve accuracy, so sampling and model-accuracy trade-offs merit further study.

    Source: §1 · p. 2

    Evaluation uses public datasets, and future work is needed to assess LLM-based parsers on logs from other systems.

    Source: §7 · p. 11

Abstract

from arXiv · show

Logs are important in modern software development with runtime information. Log parsing is the first step in many log-based analyses, that involve extracting structured information from unstructured log data. Traditional log parsers face challenges in accurately parsing logs due to the diversity of log formats, which directly impacts the performance of downstream log-analysis tasks. In this paper, we explore the potential of using Large Language Models (LLMs) for log parsing and propose LLMParser, an LLM-based log parser based on generative LLMs and few-shot tuning. We leverage four LLMs, Flan-T5-small, Flan-T5-base, LLaMA-7B, and ChatGLM-6B in LLMParsers. Our evaluation of 16 open-source systems shows that LLMParser achieves statistically significantly higher parsing accuracy than state-of-the-art parsers (a 96% average parsing accuracy). We further conduct a comprehensive empirical analysis on the effect of training size, model size, and pre-training LLM on log parsing accuracy. We find that smaller LLMs may be more effective than more complex LLMs; for instance where Flan-T5-base achieves comparable results as LLaMA-7B with a shorter inference time. We also find that using LLMs pre-trained using logs from other systems does not always improve parsing accuracy. While using pre-trained Flan-T5-base shows an improvement in accuracy, pre-trained LLaMA results in a decrease (decrease by almost 55% in group accuracy). In short, our study provides empirical evidence for using LLMs for log parsing and highlights the limitations and future research direction of LLM-based log parsers.

1 INTRODUCTION

Log parsing turns raw logs into templates by separating static text from variable values, and its accuracy matters because parsing is often the first step in log analysis. The paper proposes LLMParser, a few-shot LLM approach, and evaluates it against existing parsers.

1 INTRODUCTION

  • Parsing converts raw logs into templates by separating static text from variable values, and low parsing accuracy directly affects downstream log-analysis tasks.Logs can contain timestamps, levels, static message text, and dynamic values.Source: §1 · p. 1
  • Traditional parsers can miss log parameters, while logs’ mix of natural language and code-like variables motivates treating parsing as translation into templates.The paper suggests LLMs’ language capabilities may help with this task.Source: §1 · p. 2
  • LLMParser uses few-shot examples to teach four generative LLMs to translate logs into templates, and evaluates them on logs from 16 open-source systems.The models are Flan-T5-small, Flan-T5-base, LLaMA-7B, and ChatGLM-6B.Source: §1 · p. 2
  • LLMParsers achieve average PA 0.96 versus a highest state-of-the-art PA of 0.92, while results also show that shot size, model size, and pretraining do not yield uniformly better accuracy.Few-shot tuning outperforms in-context learning, which averages PA 0.46; Flan-T5-base is comparable to 7B LLaMA, and cross-system log pretraining has mixed results.Source: §1 · p. 2

2 BACKGROUND AND RELATED WORK

The background describes LLMs as a possible fit for logs’ mixture of natural language and code-like variables, while noting that their effectiveness for parsing remains unclear. Related work covers traditional parsers, LLM-based parsing, and LLM use in other log tasks.

2.1 Background

  • LLMs’ language and generalization capabilities may help parse logs, but the effects of differing pretraining data and model characteristics remain unclear.The authors call for a comprehensive study of LLM log parsing and the adaptations it requires.Source: §2.1 · pp. 2–3
  • In-context learning places demonstrations in each prompt, whereas few-shot tuning updates model parameters using limited examples; the paper uses tuning to incorporate log-specific knowledge.In-context demonstrations add inference cost and are limited by context size; tuning avoids repeated demonstrations and can be fast.Source: §2.1 · p. 3

2.2 Related Work

  • Traditional parsers use frequent-pattern mining, clustering, or parsing trees, but can group logs accurately while still missing variables needed for correct templates.This can hinder downstream analyses such as anomaly detection.Source: §2.2 · p. 3
  • LogPPT classifies tokens as static or variable, while ChatGPT parsing was reported less accurate than LogPPT and carries cost, fine-tuning, stability, and data-sensitivity concerns.The paper instead studies open-source text-generation and text2text LLMs that directly generate templates from prompts.Source: §2.2 · p. 3
  • Prior LLM applications include log anomaly detection and cloud-incident root-cause analysis, including LAnoBERT and the RCACopilot system.A cited Huawei Cloud case study found ChatGPT anomaly-detection effectiveness partially consistent with on-call engineers.Source: §2.2 · p. 3, Figure 2 · p. 4

3 APPROACH

LLMParser treats log parsing as a few-shot text-to-text or text-generation task, translating logs into templates. Its approach samples diverse training examples and evaluates four LLMs using model-appropriate prompts and fine-tuning.

3 APPROACH

  • The approach evaluates few-shot tuning across four research questions covering parsing, shot-count sensitivity, unseen templates, and cross-system pre-training.The stated setup samples 50 logs from each system for fine-tuning in the main evaluation.Source: §3 · p. 4

3.1 Sampling Few-Shot Data

  • The unsupervised sampler removes framework-generated data, replaces numbers, clusters logs, then samples across largest clusters until it reaches the desired size.It uses Mean Shift without requiring a preset cluster count, and labels sampled logs with their templates; the sampling strategy aims to balance coverage and diversity.Source: §3.1 · p. 4

3.2 LLMParser: Using LLMs for Parsing Logs

  • LLMParser directly translates a log message into a log template, avoiding the splitting and output-conversion steps used by LogPPT.The direct input and output are intended to make parsing results more intuitive and easier to diagnose.Source: §3.2 · p. 4
  • The evaluation compares four LLMs—Flan-T5-small, Flan-T5-base, LLaMA-7B, and ChatGLM-6B—spanning text-to-text and text-generation architectures and 80M to 7B parameters.The models are evaluated to explore how architecture and parameter size relate to parsing accuracy.Source: §3.2 · p. 4, Table 1 · p. 5
  • LLMParser uses fixed hard prompts to limit prompt variability while studying model and training-size effects.Flan-T5 uses an instruction-plus-input/output-type structure; LLaMA and ChatGLM use Alpaca’s prompt template, with log-template pairs supplied during training.Source: §3.2 · p. 5
  • LoRA fine-tunes a small additional parameter set for LLaMA-7B and ChatGLM-6B, while Flan-T5 models are fine-tuned without that optimization.The reported fine-tuning process takes from several seconds to less than five minutes.Source: §3.2 · p. 5

4 EXPERIMENT SETUP AND IMPLEMENTATION

The experiments use a corrected benchmark spanning 16 open-source systems, specify fine-tuning and generation settings, and assess both grouping and exact template parsing.

Studied Dataset

  • The corrected benchmark covers 16 open-source systems with 2,000 log messages each, spanning several software domains.Each message has a ground-truth template and parameters; the authors use the corrected dataset released by Khan et al. because prior work found labeling errors.Source: §4 · p. 5

Environment and Implementation

  • Fine-tuning used a Tesla V100, with 50-sample training taking from a few seconds to under five minutes across the studied LLMs.Single-system runs used batch size 5 for 30 epochs; cross-system runs used batch size 20 for 20 epochs.Source: §4 · pp. 5–6

Evaluation Metrics for Log Parsing

  • Parsing Accuracy requires an exact template match, whereas Group Accuracy measures correct grouping and can miss misidentified variables.The authors characterize PA as stricter and more closely aligned with practical requirements.Source: §4 · p. 6

RQ1: What is the accuracy of LLMParsers?

LLMParser evaluates four fine-tuned LLMs against established parsers across 16 systems. The LLMs achieve higher parsing accuracy overall, while results also show trade-offs involving model size, efficiency, and training-shot quantity.

RQ1: What is the accuracy of LLMParsers?

  • LLMParser_LLaMA achieved parsing accuracy (PA) of 0.9587, 4.6% higher than LogPPT’s 0.9162; overall, LLMParsers’ PA ranged from 0.9076 to 0.9587, versus Drain’s 0.3353 and Logram’s 0.1718.Source: RQ1: What is the accuracy of LLMParsers? · p. 6
  • Compared with state-of-the-art parsers, PA for LLMParser_T5Base and LLMParser_LLaMA showed a statistically significant increase (paired t-test, p-value<0.05), while GA was not significantly different from LogPPT (p-value>0.05).Source: RQ1: What is the accuracy of LLMParsers? · pp. 6–7
  • After removing training-set logs from evaluation, average PA fell by less than 1%, and LLMParser_T5Base and LLMParser_LLaMA retained average PAs of 0.9413 and 0.9426.The authors note that LogPPT’s 0.9162 PA included training data in evaluation, creating a potential data-leakage issue.Source: RQ1: What is the accuracy of LLMParsers? · p. 7
  • LLMParser_LLaMA’s accuracy advantage over LLMParser_T5Base was small, while parsing 100 Spark logs took 28.93 seconds for LLaMA versus 4 seconds for T5-base.The authors report that more complex models generally take longer to parse logs and recommend considering accuracy-efficiency trade-offs.Source: RQ1: What is the accuracy of LLMParsers? · p. 7
  • Accuracy usually stabilized or fluctuated beyond 50 training shots, although ChatGLM improved more with additional shots and sensitivity varied across models.At 100 shots, GA decreased for all models except ChatGLM, while PA improvement was barely noticeable; the authors say more shots do not always yield the best results.Source: RQ2: How does the accuracy of log… · pp. 8–9

RQ3: How is the generalizability of LLMParsers on unseen log templates?

With 50-shot tuning, LLMParsers parse unseen-template logs less accurately than seen-template logs, and those unseen logs account for a disproportionate share of errors. Similar templates can be confused, making unseen templates a bottleneck for further accuracy gains.

RQ3: How is the generalizability of LLMParsers on unseen log templates?

  • For LLMParser_LLaMA, PA was 0.638 on unseen logs versus 0.9539 on seen logs, illustrating the generalization gap.For LLMParser_T5Base, seen-log PA was 0.9506 and average unseen-log PA was 0.6385.Source: RQ3: How is the generalizability of… · p. 9
  • Similar log templates can be mistaken for one another when training data focuses on seen templates, reducing accuracy on unseen templates.The authors suggest more diverse sampling of logs and templates as a direction for improving generalization.Source: RQ3: How is the generalizability of… · pp. 9–10
  • Only 4.4% of logs had unseen templates, yet they accounted for 50% of incorrectly parsed logs; LLMParsers still outperformed Drain and Logram on unseen logs.The average PA for unseen logs was 0.3353 for Drain and 0.1718 for Logram.Source: RQ3: How is the generalizability of… · p. 9, Table 6 · p. 10

RQ4: Can pre-trained LLMParsers help improve parsing accuracy?

Pre-training on logs from other systems alone yields lower accuracy than few-shot tuning on the target system. Further target-system tuning helps T5-base but produces divergent results for LLaMA.

RQ4: Can pre-trained LLMParsers help improve parsing accuracy?

  • Pre-training on logs from 15 other systems yielded PA/GA of 0.6352/0.6313 for LLaMA and 0.7034/0.7083 for T5-base, below same-system few-shot tuning.The authors report that cross-system logs may have different characteristics.Source: RQ4: Can pre-trained LLMParsers help… · p. 10
  • After target-system tuning on 25 samples, T5-base reached GA 0.8897 and PA 0.9613, comparable to using 100 target-system samples.The 100-sample T5-base result was GA 0.89 and PA 0.98.Source: RQ4: Can pre-trained LLMParsers help… · p. 10
  • With the same 25 target-system samples, LLaMA's GA fell from 0.6352 to 0.3511, a 55% decrease, while PA rose from 0.6313 to 0.6724.The authors suggest the larger model may need more fine-tuning data.Source: RQ4: Can pre-trained LLMParsers help… · p. 10

6 DISCUSSION

LLMParsers achieve higher parsing accuracy than state-of-the-art methods, but errors remain for specific data types and unseen templates. The discussion highlights trade-offs between model complexity, efficiency, and generalization.

6 DISCUSSION

  • LLMParsers can miss specific variable types such as datetimes, even when overall parsing accuracy exceeds 90%.One example is a timestamp not recognized as a variable in the template “connection from <*> at <*>”.Source: §6 · p. 11
  • Larger models may not improve results and take more inference time, so model size must be balanced against accuracy and efficiency.The authors note that simpler models such as T5 already achieve promising parsing results.Source: §6 · p. 11
  • Unseen-template errors and confusion between similar templates motivate sampling strategies that maximize log and template diversity.The discussion also identifies target-system fine-tuning as the best-performing approach and calls for validation on more complex models such as LLaMA 70B.Source: §6 · p. 11

7 THREATS TO VALIDITY

The study's validity is bounded by its datasets, model and training-size coverage, and dependence on sampled log-template pairs. The authors describe steps taken to mitigate dataset-error and sampling concerns.

7 THREATS TO VALIDITY

  • The evaluation uses public datasets commonly used in log research, and results may be affected by log format; evaluation on other systems remains necessary.The authors use corrected datasets to mitigate potential data errors and note coverage of multiple system formats.Source: §7 · p. 11
  • The study does not cover all LLMs or training-data sizes, limiting the scope of its findings.Source: §7 · p. 11
  • The approach requires paired logs and templates, so sampling can affect results despite using an unsupervised sampling algorithm.The algorithm does not require ground-truth knowledge; the effects of sampling methods remain for future study.Source: §7 · p. 11

8 CONCLUSION

LLMParser uses clustering-sampled few-shot tuning and outperforms state-of-the-art parsers; its findings also point to training-log diversity and coverage as future research directions.

8 CONCLUSION

  • LLMParser achieves high parsing accuracy, outperforming state-of-the-art log parsers.The study evaluates models across architecture, shot-size, and pre-training settings.Source: §8 · p. 11
  • LLMParser uses few-shot tuning on a limited set of training logs sampled with a clustering algorithm.Source: §8 · p. 11
  • Compared with in-context learning, few-shot tuning achieves higher parsing accuracy and requires less inference time.Source: §8 · p. 11
  • Models may need different training-sample counts for optimal performance, motivating future study of log diversity and coverage rather than simply increasing sample size.Source: §8 · p. 11

Related papers

Loading 2404.18001v1…