PaperlayerView exact revision on arXiv

Source-linked AI summary

Log Parsing with Prompt-based Few-shot Learning

Van-Hoang Le, Hongyu Zhang

arXiv:2302.07435v1cs.SE

TL;DR

Existing log parsers can miss the semantic distinction between keywords and parameters and often need dataset-specific knowledge. LogPPT applies prompt-based few-shot learning to labelled logs selected by adaptive sampling; across 16 public datasets, it reports accuracy above 0.9 and outperforms state-of-the-art parsers. Its evaluation uses public benchmarks whose ground-truth templates may contain some errors.

  • Problem

    Existing parsers overlook log-message semantics and require domain-specific knowledge, limiting their ability to identify correct templates and parameters consistently across datasets.

    Source: §I · pp. 1–2

  • Method

    LogPPT uses prompt-based few-shot learning with a pretrained language model and an adaptive random sampling algorithm to select a small, diverse labelled training set.

    Source: §I · p. 2

  • Results

    With 32 labelled samples, LogPPT achieves over 0.9 average Group Accuracy and Parsing Accuracy across 16 public datasets, outperforming state-of-the-art parsers by 16% on Group Accuracy and about 84% on Parsing Accuracy.

    Source: §I · p. 2

  • Takeaways & Limitations

    LogPPT requires no manually defined preprocessing regular expressions and uses the same hyperparameter values across datasets, allowing it to adapt to new log datasets.

    Source: §I · p. 2

    The evaluation relies on public datasets whose ground-truth templates may contain a small proportion of errors, although the authors use the latest corrected benchmark versions.

    Source: B. Threats to Validity · p. 10

Abstract

from arXiv · show

Logs generated by large-scale software systems provide crucial information for engineers to understand the system status and diagnose problems of the systems. Log parsing, which converts raw log messages into structured data, is the first step to enabling automated log analytics. Existing log parsers extract the common part as log templates using statistical features. However, these log parsers often fail to identify the correct templates and parameters because: 1) they often overlook the semantic meaning of log messages, and 2) they require domain-specific knowledge for different log datasets. To address the limitations of existing methods, in this paper, we propose LogPPT to capture the patterns of templates using prompt-based few-shot learning. LogPPT utilises a novel prompt tuning method to recognise keywords and parameters based on a few labelled log data. In addition, an adaptive random sampling algorithm is designed to select a small yet diverse training set. We have conducted extensive experiments on 16 public log datasets. The experimental results show that LogPPT is effective and efficient for log parsing.

I. INTRODUCTION

Logs support system understanding and diagnosis, and parsing structures raw messages for downstream analytics. LogPPT addresses parsers’ semantic and dataset-specific limitations with prompt-based few-shot learning and reports strong results across 16 public datasets.

I. INTRODUCTION

  • Log parsing converts raw log messages into templates and runtime parameters, structuring data for downstream analytics such as anomaly detection and root-cause analysis.Logs also support failure prediction and log compression.Source: §I · p. 1
  • Existing parsers use statistical features and can mistake parameters for keywords because they overlook log-message semantics.They also require dataset-specific preprocessing and hyperparameter settings.Source: §I · p. 1
  • LogPPT uses prompt-based few-shot learning to recognize keywords and parameters from a small labelled set selected by Adaptive Random Sampling.It tunes a pretrained language model to predict the virtual label token “PARAM”; it requires no preprocessing and uses the same hyperparameter values across datasets.Source: §I · p. 2
  • With 32 labelled samples, LogPPT achieves over 0.9 average Group Accuracy and Parsing Accuracy, outperforming state-of-the-art parsers by 16% on Group Accuracy and about 84% on Parsing Accuracy.The evaluation covers 16 public log datasets and reports the method as effective and efficient.Source: §I · p. 2

A. Log Parsing

Log parsing extracts static templates and dynamic parameters from raw logs. Traditional methods use pattern mining, clustering, or heuristics, while pretrained-model fine-tuning reframes parsing as token classification but depends on downstream data scale.

A. Log Parsing

  • Data-driven parsing approaches include frequent-pattern mining, similarity-based clustering, and heuristics-based parsing.Examples include SLCT, LFA, and Logram; LKE, LogSig, and LenMa; and AEL, Spell, and Drain, respectively.Source: A. Log Parsing · p. 2
  • Traditional log parsers can misclassify semantically meaningful parameters as keywords and require dataset-specific regular expressions and hyperparameter adjustments.For example, a regular expression for HDFS block IDs must be changed to match BGL core IDs.Source: A. Log Parsing · pp. 2–3

1) Pre-training and Fine-tuning:

  • Fine-tuning reframes log parsing as token classification, labeling each token as a keyword or parameter with an added classifier.The mismatch between pretraining and classification objectives can limit use of pretrained knowledge, and performance depends significantly on downstream data scale.Source: 1) Pre-training and Fine-tuning · p. 3

2) Prompt Tuning:

Prompt tuning recasts log parsing as masked-label prediction, while LogPPT uses template-free prompting to avoid manually designed templates and process tokens in one pass. The approach combines this tuning with few-shot learning and adaptive selection of labelled samples.

2) Prompt Tuning:

  • For log parsing, a prompt template supplies the message, target token, and label slots; a verbalizer maps predicted label words to keyword or parameter classes.Enumerating tokens yields the log template and parameters.Source: 2) Prompt Tuning · p. 3
  • Hard prompts use fixed natural-language instructions, whereas soft prompts use learnable continuous virtual-token vectors.Hard-prompt template and label-word choices require task-specific knowledge.Source: 2) Prompt Tuning · p. 3
  • Standard prompt tuning requires enumerating span candidates and is sensitive to noise.Source: 2) Prompt Tuning · p. 3
  • Template-free prompting replaces prompt templates with a generated virtual label token that the model predicts at parameter positions while retaining original tokens for keywords.The method uses a custom masked-language-model objective.Source: 2) Prompt Tuning · p. 3
  • Template-free prompting processes all tokens simultaneously in one-pass decoding, avoiding standard prompting’s candidate enumeration.Source: 2) Prompt Tuning · pp. 3–4

III. APPROACH

  • LogPPT applies prompt tuning to few-shot log parsing and selects a small, optimized labelled training set with Adaptive Random Sampling.Source: §III · p. 4

A. Problem Definition

LogPPT frames parsing as recognizing parameters in context: the model predicts a virtual PARAM token for parameters and original tokens for keywords. Its sampling procedure selects a small, diverse set of logs for labelling and training.

A. Problem Definition

  • The model predicts PARAM at parameter positions and preserves the original token at keyword positions, turning log parsing into parameter recognition.During inference, PARAM predictions are parameters and the other tokens form the template.Source: A. Problem Definition · p. 4
  • PARAM is a virtual token without linguistic meaning, with an embedding derived from frequent parameters using labelled and unlabelled data.Source: A. Problem Definition · p. 4

B. Few-shot Data Sampling

  • Adaptive sampling starts with the shortest cleaned log, then iteratively adds a candidate selected for low similarity to its nearest neighbour in the current sample set.The algorithm draws random candidates and repeats until the set contains K samples.Source: B. Few-shot Data Sampling · pp. 4–5
  • Selected original log messages receive templates from user feedback and are returned as the K-labelled training set.Source: B. Few-shot Data Sampling · p. 5

C. Prompt-Tuning for Log Parsing

LogPPT uses an entity-oriented language-model objective to distinguish keywords from parameters in log messages. It predicts a shared PARAM label for dynamic parameters and the original words for keywords.

1) Pre-trained Language Model:

LogPPT uses RoBERTa as its pretrained language model and generates a virtual label token to represent parameters. Candidate token predictions and their frequency in unlabelled data inform that token’s embedding.

1) Pre-trained Language Model:

  • RoBERTa’s byte-level BPE can tokenize out-of-vocabulary developer-created parameters into subwords, avoiding unknown tokens.The authors choose RoBERTa partly because parameters are uncommon and constantly changing.Source: 1) Pre-trained Language Model · p. 5

2) Virtual Label Token Generation:

3) Training:

LogPPT trains a language model to identify template keywords and parameters, then parses each message in one decoding pass. The evaluation is organized around effectiveness, efficiency, module contributions, and tuning techniques.

3) Training:

  • Training replaces parameter positions with the virtual label “PARAM” in a target sequence, then tunes the language model to predict that sequence from each input message.The objective reuses the pretrained model to retain its learned knowledge; K denotes the number of labelled training samples.Source: 3) Training · pp. 5–6

D. Online Parsing

  • Online parsing tokenizes each message, predicts whether each token is “PARAM,” and assembles parameter lists and templates before merging consecutive parameters.The process uses one-pass decoding.Source: D. Online Parsing · p. 6

A. Research Questions

  • The evaluation asks whether LogPPT is effective and efficient, how its modules contribute, and how it performs with different tuning techniques.These are the four stated research questions.Source: A. Research Questions · p. 6

B. Datasets

The experiments use corrected annotations from 16 diverse public log datasets and compare LogPPT with five established parsers. They assess parsing quality with three complementary metrics and report implementation settings.

B. Datasets

  • The evaluation uses corrected versions of 16 LogPai datasets spanning distributed systems, supercomputers, operating systems, mobile systems, server applications, and standalone software.Earlier annotations were manually reviewed, and subsequent work applied heuristic rules to correct errors.Source: B. Datasets · p. 6

C. Baselines

  • LogPPT is compared with AEL, LenMa, Spell, Drain, and Logram using publicly available implementations and labelled data extended to each baseline.The selected parsers were identified in prior work as highly accurate and efficient among evaluated parsers.Source: C. Baselines · p. 6

D. Evaluation Metrics

  • Group Accuracy measures whether messages are grouped consistently with ground truth, Parsing Accuracy requires every token to be correctly classified, and Edit Distance measures template string similarity.Parsing Accuracy is stricter than Group Accuracy; Edit Distance is the median distance between parsed and ground-truth templates.Source: D. Evaluation Metrics · p. 6

E. Implementation and Environment

  • Experiments use AdamW with a 5e−5 initial learning rate, batch size 8, and 200 training steps; online parsing uses batch size 32 and the default K is 32.The virtual label embedding uses the eight most frequent label tokens; results remain consistently good with at least four.Source: E. Implementation and Environment · p. 7

V. EXPERIMENTAL RESULTS

With 32 labelled samples, LogPPT outperforms baseline parsers on nearly all datasets across the reported metrics. Its accuracy is strong across log types, and it maintains good performance with fewer labelled samples.

1) Accuracy:

  • LogPPT achieves an average Edit Distance of 1.130, seven times better than Drain, and beats baselines on 15 of 16 datasets.On Apache, its result is comparable to Drain’s: 0.024 versus 0.Source: 1) Accuracy · p. 7

2) Robustness:

  • Across datasets, LogPPT has the smallest accuracy variance and median robustness values of 0.99 for GA and 0.94 for PA, exceeding Drain by 6.9% and 98.5%, respectively.It uses the same hyperparameter values across datasets and needs no manually defined regular expressions.Source: Figure 4 · p. 7, 2) Robustness · p. 7

1) Accuracy:

  • LogPPT reaches 0.923 average GA versus Drain’s 0.797 (15.8% higher) and 0.916 average PA, at least 83.9% above baselines.It has the best GA on 12 of 16 datasets and the best PA on 14 of 16.Source: 1) Accuracy · p. 7, Table I · p. 8

2) Robustness:

  • LogPPT remains consistently good when K ≥16 and outperforms baselines on Parsing Accuracy and Edit Distance with only four labelled samples.Performance drops severely when fewer training samples are used.Source: 2) Robustness · pp. 7–8, Figure 5 · p. 8

3) Accuracy with Unseen Logs:

LogPPT parses previously unseen log events more accurately than the baselines and scales to large log volumes with GPU acceleration. Ablations also show worse parsing accuracy and edit distance when either key module is removed.

3) Accuracy with Unseen Logs:

  • On unseen log data, LogPPT achieves Parsing Accuracy of 0.599, exceeding existing parsers by 58.9% to 517.5%.The evaluation treats events appearing only once in a dataset as unseen; there are 42.64 such events on average across the 16 datasets.Source: 3) Accuracy with Unseen Logs · p. 8, Table II · p. 8

B. RQ2: Runtime Performance Evaluation

  • At one million messages, LogPPT takes about 107 seconds, slightly slower than Drain (94s), Spell (95s), and AEL (84s), but faster than LenMa and Logram, which exceed 1,000 seconds.Its running time increases slowly as log volume increases, and GPU acceleration makes it faster than or comparable to traditional parsers.Source: B. RQ2: Runtime Performance Evaluation · pp. 8–9, Figure 6 · p. 9

C. RQ3: Ablation Study

  • Without Virtual Label Token Generation, Parsing Accuracy falls to 0.835, 8.8% below complete LogPPT; removing Adaptive Random Sampling decreases Parsing Accuracy by 23.1%.The first module helps identify parameter-representative pivot words, while naive sampling can yield low-quality training data for imbalanced log templates.Source: C. RQ3: Ablation Study · p. 9, Table III · p. 9

D. RQ4: Comparison with Different Tuning Techniques

LogPPT’s prompt tuning outperforms the compared tuning methods, using both semantic and positional information to parse log tokens. It also has parsing time similar to fine-tuning, while other prompt methods require token enumeration.

D. RQ4: Comparison with Different Tuning Techniques

A. Why does LogPPT Work?

  • LogPPT does not require dataset-specific regular expressions, making it easier to apply to a new log dataset.Its prompt-tuning method also avoids complex prompt-instruction design.Source: A. Why does LogPPT Work? · p. 10
  • LogPPT’s semantic and positional cues distinguish tokens with the same text but different roles, such as the two “proxy” tokens in a Proxifier message.Standard prompt tuning predicts the same label for both tokens, while the example reports 100% parsing accuracy with LogPPT.Source: A. Why does LogPPT Work? · p. 10

B. Threats to Validity

The evaluation uses public log datasets whose ground-truth templates may contain some errors, and compares against related approaches using their replication packages and previously optimized settings. The paper concludes that LogPPT performs effectively on public datasets, while production scalability and effectiveness remain future evaluation goals.

B. Threats to Validity

  • The public evaluation datasets may contain a small proportion of ground-truth template errors, despite using the latest corrected benchmark versions.The benchmark ground truth includes log templates and corresponding parameters.Source: B. Threats to Validity · p. 10
  • The comparison uses related approaches’ replication-package implementations and parameters optimized in prior work.The settings include such parameters as the number of log templates and similarity threshold.Source: B. Threats to Validity · p. 10

VII. RELATED WORK

  • Prior token-classification parsers include LogStamp, which uses pseudo-labels from a traditional parser, and UniParser, which needs labelled data across sources and handcrafted token-splitting rules.The passage notes that pseudo-labels can introduce training noise and that UniParser’s requirements may limit practical availability or dataset suitability.Source: §VII · p. 11

VIII. CONCLUSION

  • LogPPT uses a pre-trained language model and few labelled examples without domain-specific preprocessing knowledge, and the conclusion reports better performance than state-of-the-art parsers on public datasets.The conclusion identifies production-environment evaluation of scalability and effectiveness as future work.Source: §VII · p. 11, §VIII · p. 11

Related papers

Loading 2302.07435v1…