PaperlayerView exact revision on arXiv

Source-linked AI summary

LILAC: Log Parsing using LLMs with Adaptive Parsing Cache

Zhihan Jiang, Jinyang Liu, Zhuangbin Chen, Yichen Li, Junjie Huang, Yintong Huo, Pinjia He, Jiazhen Gu, Michael R. Lyu

arXiv:2310.01796v3cs.SE

TL;DR

Existing parsers can struggle with complicated logs, while LLM-based parsing faces accuracy, consistency, and overhead challenges. LILAC combines ICL-selected demonstrations with an adaptive parsing cache. It reports improved benchmark accuracy and efficiency compared with state-of-the-art methods.

  • Problem

    Existing log parsers can struggle with complicated logs, and LLMs bring specialization, consistency, and overhead challenges to practical log parsing.

    Source: §1 · p. 2, §2.3 · p. 5

  • Method

    LILAC combines ICL-based parsing with hierarchical candidate sampling, high-quality demonstrations, and an adaptive cache that stores and refines templates.

    Source: §1 · p. 3, §8 · p. 20

  • Results

    69.5% outperformance over state-of-the-art baselines for template-accuracy F1; LILAC reports highest average accuracy across metrics and speed comparable to Drain.

    Source: §1 · p. 3, §8 · p. 20

  • Takeaways & Limitations

    LILAC's cache reuses parsed templates to reduce duplicate LLM queries and adaptively refines stored templates to mitigate inconsistency.

    Source: §1 · p. 3, §3.3.1 · p. 10

    The authors consider data leakage negligible in their experiments, noting that the GPT-3.5-turbo-0613 updates stopped before Loghub-2.0 ground-truth templates were public.

    Source: §6.2 · p. 19

Abstract

from arXiv · show

Log parsing transforms log messages into structured formats, serving as the prerequisite step for various log analysis tasks. Although a variety of log parsing approaches have been proposed, their performance on complicated log data remains compromised due to the use of human-crafted rules or learning-based models with limited training data. The recent emergence of powerful large language models (LLMs) demonstrates their vast pre-trained knowledge related to code and logging, making it promising to apply LLMs for log parsing. However, their lack of specialized log parsing capabilities currently hinders their accuracy in parsing. Moreover, the inherent inconsistent answers, as well as the substantial overhead, prevent the practical adoption of LLM-based log parsing. To address these challenges, we propose LILAC, the first practical log parsing framework using LLMs with adaptive parsing cache. To facilitate accurate and robust log parsing, LILAC leverages the in-context learning (ICL) capability of the LLM by performing a hierarchical candidate sampling algorithm and selecting high-quality demonstrations. Furthermore, LILAC incorporates a novel component, an adaptive parsing cache, to store and refine the templates generated by the LLM. It helps mitigate LLM's inefficiency issue by enabling rapid retrieval of previously processed log templates. In this process, LILAC adaptively updates the templates within the parsing cache to ensure the consistency of parsed results. The extensive evaluation on public large-scale datasets shows that LILAC outperforms state-of-the-art methods by 69.5% in terms of the average F1 score of template accuracy. In addition, LILAC reduces the query times to LLMs by several orders of magnitude, achieving a comparable efficiency to the fastest baseline.

1 INTRODUCTION

LILAC addresses limits in existing log parsers and practical barriers to using LLMs by combining ICL-based parsing with an adaptive cache. Its evaluation reports improved accuracy and efficiency against baselines.

1 INTRODUCTION

  • Existing syntax-based parsers rely on crafted rules, while learning-based parsers have limited labeled data and may struggle with complicated logs.These limits compromise their performance on diverse or unfamiliar log data.Source: §1 · p. 2
  • LLMs may lack specialized log-parsing capability, produce inconsistent templates, and impose high inference and network overhead.Inconsistent templates for messages with the same template can lower grouping accuracy, while inference costs challenge processing large log volumes.Source: §1 · pp. 2–3
  • LILAC uses ICL demonstrations selected through candidate sampling and demonstration selection, and an adaptive cache that stores and refines generated templates.Cache matching avoids duplicate LLM queries; cache updating refines potential errors to mitigate inconsistent outputs.Source: §1 · p. 3
  • 69.5% for template-accuracy F1 and 66.8% for grouping F1 over state-of-the-art baselines; LILAC's speed is comparable to Drain.The evaluation uses public, large-scale Loghub-2.0 datasets.Source: §1 · p. 3

2 BACKGROUND AND MOTIVATION

The background reviews limits of existing parsers, explains in-context learning as an LLM adaptation strategy, and identifies practical challenges for LLM-based parsing.

2.1 Log Parsing

  • Syntax-based parsers depend on designed rules and features, while semantic-based parsers rely on labeled training data; both can struggle with complex, evolving logs.Rule-based methods can fail when log parameters vary in length, and training-based methods may degrade as production logs change.Source: §2.1 · p. 4

2.2 Large Language Models

  • ICL prompts combine task instructions, query-answer demonstrations, and a query to convey task-specific input-output relationships without model fine-tuning.The paper adopts ICL to use LLMs for log parsing.Source: §2.2 · pp. 4–5

2.3 Challenges of Log Parsing with LLMs

  • The paper identifies unspecialized LLMs, inefficient candidate sampling, inconsistent templates, and costly per-message queries as practical adoption barriers.Diverse candidate sampling is challenging at scale; inconsistent templates can reduce grouping accuracy, while query overhead makes high-volume parsing difficult.Source: §2.3 · p. 5

3 METHOD

LILAC combines cache-based reuse and refinement with an ICL-enhanced parser that samples diverse candidates, selects similar demonstrations, and prompts an LLM to generate templates.

3.1 Overview

  • The cache first seeks an exact template match; on a miss, it retrieves relevant templates, invokes the ICL-enhanced parser, and updates the cache using the generated and relevant templates.Cache hits avoid duplicate LLM queries; cache updates may insert a template or refine a relevant one.Source: §3.1 · p. 6

3.2 ICL-enhanced parser

  • The ICL-enhanced parser samples diverse candidate messages, selects query-similar demonstrations, and places them in a designed prompt for the LLM.This uses ICL to adapt the LLM to log parsing without resource-intensive model tuning.Source: §3.2 · pp. 6–7

3.2.1 Candidate Sampling.

  • Candidate sampling clusters messages first by frequent tokens and then by special-character formats, before distributing quotas across clusters to favor diversity and representation.Larger fine-grained clusters receive priority when quota distribution cannot be even; assigned candidates are selected randomly.Source: §3.2.1 · pp. 7–8

3.2.2 Demonstration Selection.

  • For each query, LILAC selects the top-k candidates by Jaccard similarity over token-derived characters and special characters.The selected messages provide examples with similar tokens and formats to the query.Source: §3.2.2 · p. 8

3.2.3 Query Design.

  • The prompt combines task instructions, log-message/template demonstrations, and the queried log; similar demonstrations are used to guide template generation.The examples are arranged with higher-similarity demonstrations closer to the queried log.Source: §3.2.3 · p. 9

3.3 Adaptive Parsing Cache

LILAC stores generated templates in a tree for retrieval, then uses cache matching and adaptive updates to avoid duplicate LLM queries and improve template consistency.

3.3 Adaptive Parsing Cache

  • The cache tree stores tokenized templates, enabling retrieval by a single root-to-leaf traversal and filtering related templates through shared prefixes.Each leaf represents a template; “<*>” is a wildcard that can match variable-length token sequences.Source: §3.3 · p. 9

3.3.1 Cache Matching.

  • Cache matching checks whether an input log's template is stored; an exact leaf match returns its template ID without another LLM query.If matching stops at internal nodes instead, their subtrees supply relevant templates for a later update.Source: §3.3.1 · pp. 10–11

3.3.2 Cache Updating.

  • When matching fails, the ICL parser generates a template that LILAC compares with relevant cached templates to distinguish a new pattern from an inconsistent parse.If the highest similarity is below the threshold, LILAC inserts a new template; above it, differing tokens are replaced with “<*>” to refine the cached template. Updating overhead is typically minimal.Source: §3.3 · pp. 9–10, §3.3.2 · p. 11

4 EXPERIMENTAL SETUP

The evaluation tests LILAC’s effectiveness, design, model generalizability, and efficiency on large-scale log datasets using established parser baselines and multiple accuracy metrics.

4.1 Research Questions

  • The evaluation asks how effective and efficient LILAC is, how its designs contribute, and how it performs with different LLMs.Source: §4.1 · p. 11

4.2 Datasets and baselines

  • Loghub-2.0 supplies 14 labeled datasets averaging 3.6 million messages each; four open-source parsers provide syntax-based and semantic-based baselines.AEL and Drain represent syntax-based methods; UniParser and LogPPT represent semantic-based methods.Source: §4.2 · p. 12

4.3 Metrics

  • Four metrics assess message grouping and parsing accuracy, with F1 variants evaluating correctness at the template level.Grouping Accuracy and Parsing Accuracy are message-level measures; FGA and FTA are template-level measures.Source: §4.3 · p. 12

4.4 Implementation and Environment

  • Experiments use ChatGPT by default, sample candidates from each dataset’s first 20% of messages, and report medians across five repetitions for randomized runs.The default sample and demonstration counts are 32 and 3, respectively.Source: §4.4 · pp. 12–13

5.1 RQ1: How effective is LILAC in parsing log messages?

Across public datasets, LILAC leads the baselines on average accuracy metrics and shows lower performance variance. Its reported advantages cover grouping, template identification, and robustness across diverse log data.

5.1.1 Accuracy.

  • 66.8% FGA improvement over Drain and 69.5% FTA improvement over LogPPT accompany LILAC’s lead over all baselines on average metrics.Source: Table 1 · p. 13, §5.1.1 · p. 13, §5.1.2 · p. 14
  • 92.7% GA and 92.4% FGA are LILAC’s average scores, compared with Drain’s 84.3% GA and 55.4% FGA.Source: §5.1.1 · p. 13
  • 84.2% PA and 81.0% FTA for LILAC exceed LogPPT’s 73.6% PA and 47.8% FTA by 14.4% and 69.5%, respectively.Source: §5.1.1 · pp. 13–14

5.1.2 Robustness.

  • LILAC has the narrowest performance distribution; its standard deviations for GA, FGA, PA, and FTA are 9.3%, 6.9%, 15.2%, and 12.9%, versus LogPPT’s 26.0%, 27.4%, 27.6%, and 27.3%.The authors attribute robustness to LLM pre-trained log knowledge and ICL adaptation to dataset-specific characteristics.Source: Figure 5 · p. 14, §5.1.2 · p. 14

5.2 RQ2: How does each design contribute to LILAC?

RQ2 evaluates the contributions of LILAC’s ICL-enhanced parser and adaptive parsing cache. The results report gains from both designs, with ICL performance also varying by candidate and demonstration settings.

5.2.1 ICL-enhanced Parser.

  • Removing ICL decreases average FTA by 27.9%; random sampling and random demonstration selection decrease it by 17.7% and 15.7%, respectively.Replacing the candidate sampler with LogPPT’s algorithm decreases average FTA by 14.2%.Source: §5.2.1 · p. 15, Table 2 · p. 15
  • With three demonstrations, average FTA is approximately 75%, 81%, and 85% for 8, 32, and 128 candidates; 32 candidates and 3 demonstrations are the default.Performance is lowest with one demonstration and fluctuates toward stability above three; all tested settings outperform LILAC without ICL.Source: §5.2.1 · pp. 15–16, Figure 6 · p. 15

5.2.2 Parsing Cache.

  • Adding the parsing cache raises LogPPT’s mean GA, FGA, PA, and FTA by 48.4%, 55.1%, 6.0%, and 41.0%, respectively.The authors attribute grouping gains to cache matching and adaptive updates, and template-accuracy gains to refinement using historical templates.Source: §5.2.2 · p. 16, Table 3 · p. 16

5.2 RQ2: How does each design contribute to LILAC?

  • Both the ICL-enhanced parser and parsing cache contribute to LILAC’s overall performance, addressing parsing accuracy and language-model output inconsistency, respectively.The ICL strategies support accurate parsing, while the cache mitigates inconsistency in language models.Source: §5.2.2 · p. 16

5.3 RQ3: How capable is LILAC integrated with different LLMs?

LILAC maintains high performance across the evaluated LLMs, though performance varies with model capabilities. The smaller Curie model remains above existing log parsers in accuracy.

5.3 RQ3: How capable is LILAC integrated with different LLMs?

  • The evaluation uses ChatGPT and Davinci, each with 175B parameters, and Curie, with 13B parameters.ChatGPT is fine-tuned for conversational tasks, while Davinci has enhanced text-generation capabilities.Source: §5.3 · pp. 16–17
  • Davinci’s average FTA exceeds ChatGPT’s by 0.6%, while Curie’s average FTA is 12.1% lower than ChatGPT’s.The authors suggest Davinci’s text-generation focus aligns with log parsing; they attribute Curie’s lower performance to its smaller parameter size and more limited pre-training knowledge.Source: §5.3 · p. 17
  • LILAC consistently achieves high performance across ChatGPT, Davinci, and Curie, including with the relatively smaller Curie model.LILAC with Curie still surpasses existing log parsers.Source: §5.3 · p. 17, Table 4 · p. 17

5.4 RQ4: How efficient is LILAC in processing large-scale log data?

RQ4 assesses candidate-sampling and parsing efficiency on public Loghub-2.0 datasets. LILAC’s parsing efficiency is comparable to Drain and exceeds the reported semantic-based baselines.

5.4.1 Candidate Sampling Efficiency.

  • For 32 candidates, LILAC sampling takes 19.3 seconds versus more than 1300 seconds for LogPPT, a 67.6 speedup.LILAC’s sampling time remains stable as candidate counts increase, whereas LogPPT’s increases linearly.Source: §5.4.1 · pp. 17–18, Table 5 · p. 17

5.4.2 Parsing Efficiency.

  • LILAC takes 569.6 seconds to parse an average of 3.6 million messages, versus Drain’s 425.4 seconds; cache operations account for 376.5 seconds and parser queries for 193.1 seconds.LILAC averages 279.7 LLM queries versus 249 ground-truth templates, compared with over 3.6 million queries for existing LLM-based approaches.Source: §5.4.2 · p. 18
  • LILAC reduces LLM queries by matching the parsing cache to prevent duplicates, surpassing semantic-based methods by 4.03 to 7.19 times and remaining comparable to the fastest syntax-based methods.Fig. 7 presents the efficiency comparison among baselines and LILAC on large-scale datasets.Source: §5.4.2 · p. 18, Figure 7 · p. 18

6 DISCUSSION

The discussion presents LILAC as a framework for production log parsing that combines an adaptive cache with candidate sampling and demonstration selection. It also identifies privacy and manual annotation as practical considerations.

6.1 Practicality of LILAC

  • LILAC uses an adaptive cache to reduce LLM query costs and instability, and candidate sampling and demonstration selection to support ICL.The authors state that its per-query token count is generally small and that it is compatible with traditional language models such as RoBERTa.Source: §6.1 · p. 19

6.2 Threats to Validity

  • External LLM processing may pose privacy and security risks for sensitive enterprise logs; integrating users’ own models is presented as an avoidance option.ICL requires manual annotation, though the authors report improved performance with a small labeled set such as 32 messages.Source: §6.2 · p. 19

7 RELATED WORK

Related work divides log parsers into syntax-based and semantic-based approaches, while recent LLM-based methods still face scalability, efficiency, and consistency challenges. LILAC combines an ICL-enhanced parser with an adaptive cache to address these challenges.

7 RELATED WORK

  • Syntax-based parsers use frequency, similarity, or heuristic features to distinguish templates from parameters in log messages.The related-work overview groups syntax-based parsers into these three categories.Source: §7 · p. 19
  • Semantic-based parsers can achieve higher parsing accuracy by mining message semantics, but typically require labeled log data for training or tuning.Examples include token-classification models and LogPPT, which tunes a pre-trained language model.Source: §7 · p. 20
  • Earlier LLM-based methods either sample demonstrations using pairwise distances across log-message embeddings, which is infeasible at enormous scale, or parse each log in a single query without addressing efficiency and consistency.LILAC is presented as combining an ICL-enhanced parser with an adaptive parsing cache to address these issues.Source: §7 · p. 20

8 CONCLUSION

LILAC combines ICL-based demonstration selection with an adaptive parsing cache for practical LLM log parsing. Experiments on large-scale datasets show it outperforms state-of-the-art baselines with high efficiency.

8 CONCLUSION

  • LILAC outperforms all state-of-the-art baselines with high efficiency in extensive experiments on large-scale log datasets.The conclusion reports this overall result without specifying a metric or numerical margin.Source: §8 · p. 20
  • Candidate sampling and demonstration selection use LLM in-context learning to adapt parsing across varied log data.The algorithms select high-quality demonstrations.Source: §8 · p. 20
  • Cache matching and adaptive updating store log templates and help mitigate LLM inconsistency and inefficiency.The adaptive parsing cache stores templates generated during parsing.Source: §8 · p. 20

Related papers

Loading 2310.01796v3…