Interpretable Online Log Analysis Using Large Language Models with Prompt Strategies

Motivation

• Prediction-only log analysis leaves analysts without justification for interpreting results or deciding what action to take.Log parsing and anomaly detection help summarize raw logs and identify unusual events, respectively.Source: §1 · p. 2

Contributions

• LogPrompt combines self-prompt, chain-of-thought, and in-context strategies for online log parsing and anomaly detection.The authors report state-of-the-art performance without in-domain training and useful justifications for predictions.Source: §1 · p. 2
• Up to 55.9%: LogPrompt outperforms approaches trained on thousands of in-domain logs without requiring substantial in-domain training.The evaluation spans nine public datasets and log parsing and anomaly detection tasks.Source: §1 · p. 2
• Six senior practitioners rated more than 80% of generated explanations for variables and anomalies highly useful and readable.The explanations could help O&M engineers understand program status, compose reports, and rule out false alarms.Source: §1 · p. 2

2.1 Large Language Models

• LLMs predict text autoregressively, and prompts can activate their capabilities and improve performance on tasks.The text describes LLM training as estimating the next word from the preceding sequence.Source: §2.1 · p. 2

2.2 Pre-training & Prompting

• Pre-training and prompting guides an LLM with task-specific text without changing its parameters, unlike fine-tuning, which adds and trains task-specific parameters.A prompt comprises a fixed task prefix, an input slot [X], and an answer slot [Z].Source: §2.2 · pp. 2–3

3.1 Motivation

• 380.7% performance fluctuation across prompt strategies, alongside LogRobust F1 falling from 0.88 to 0.33 as training data shrinks, motivates online-adapted prompting.Existing methods perform nearly at 100% with 80% or 90% training data but deteriorate when training data is 10% or less.Source: §3.1 · pp. 3–4, Figure 3 · p. 3

3.2 Overview of LogPrompt

• LogPrompt standardizes input and answer formats, then assembles them with a tailored prompt prefix to obtain an LLM response.Its prefix strategies are self-prompt, chain-of-thought, and in-context prompting.Source: §3.2 · p. 4

3.4 Prompt Strategies

• LogPrompt constructs prompt prefixes through self-prompt, chain-of-thought, and in-context strategies, each based on a distinct prompting principle.The strategies respectively generate prompt candidates, structure intermediate reasoning, and provide task examples.Source: §3.2 · p. 4

3.3 Format Controls

• Format controls organize multiple input logs and constrain answers to a per-log format, addressing omitted logs, inconsistent formats, and unrelated output.The answer format specifies an allowed answer range and a reason for each log.Source: §3.3 · p. 4

3.4.1 Self-prompt.

• 18.4%: Prompt 2 outperformed Prompt 5 in log parsing; the authors selected Prompt 2, noting that accurate wording and stepwise instructions tended to perform better.The candidates were assessed on the first 100 Android logs using the average of RandIndex and F1-score.Source: §3.4.1 · pp. 4–5, Figure 5 · p. 5

3.4.2 CoT Prompt.

• The chain-of-thought prompt requests reasons and constrains intermediate steps, including treating only explicit textual alerts as abnormal and ignoring wildcards.These constraints respond to simple-prompt cases where ChatGPT inferred anomalies from invalid-looking values or missing information.Source: §3.4.2 · p. 5

3.4.3 In-context Prompt.

• In-context prompting supplies a few labeled logs to establish task context without iterative parameter training; parsing examples pair each log with its template.For anomaly detection, the examples show normal and abnormal categories.Source: §3.4.3 · p. 5

• The experiments are organized around a setup, four research questions, and discussion of their results.Source: §4 · p. 6

4.1 Experimental Setup

• Table 1 reports log-dataset statistics, and “!” marks datasets used as task evaluation sets with human labels.Source: Table 1 · p. 6, §4.1 · p. 6

4.1.1 Datasets.

• The evaluation spans nine domains; two datasets have expert-annotated anomalies, while eight have manually extracted templates from 2,000 messages each.All log data are timestamped, enabling chronological train/test splits.Source: §4.1.1 · p. 6

4.1.2 Implementation Details.

• Primary experiments use ChatGPT via OpenAI APIs, with an open-source smaller LLM also deployed using one NVIDIA A100 GPU.Chronological splits simulate online scenarios; invalid response formats are retried, and format failure is reported below 1%.Source: §4.1.2 · p. 6

4.2.1 RQ1: How effective is LogPrompt compared with existing methods in the online scenario, when being faced with scarce training data?

• LogPrompt achieved the best F1-score on six of eight parsing datasets and outperformed LogPPT by 32.8% and LogStamp by 37.4% on average; anomaly detection improved by 55.9% on average without in-domain training.The parsing comparisons use Table 2, and the anomaly-detection results use Table 3.Source: §4.2.1 · pp. 6–7, Table 2 · p. 7, Table 3 · p. 7

4.2.2 RQ2: Does LogPrompt’s interpretability yield helpful and comprehensible content for practitioners during real-world log analysis?

• Reviewers’ average usefulness and readability scores exceeded four, and average high-interpretability percentages exceeded 80% across evaluation dimensions.Six practitioners with over ten years of O&M experience rated 200 samples; high interpretability denotes scores higher than four.Source: §4.2.2 · pp. 7–8
• Practitioner comments describe explanations as useful for quickly understanding unfamiliar-domain logs and communicating analysis after crashes.A reviewer also said explanations can help engineers assess anomaly credibility more promptly.Source: §4.2.2 · p. 8
• Low-rated explanations often overgeneralized domain-specific terms or struggled with logs that were brief or rich in non-NLP patterns.The authors identify missing domain knowledge and limited semantic content as contributing factors.Source: §4.2.2 · p. 8

4.2.3 RQ3: How do varying prompt strategies within LogPrompt impact the performance of LLMs in log analysis tasks?

• For BGL, adding chains of thought improved session-level F1-score by 0.195 and template-level F1-score by 0.206.The study successively added implicit and explicit CoT modules to the simple prompt.Source: §4.2.3 · p. 9, Table 7 · p. 9
• With only implicit CoT, BGL session-level F1-score improved by 62.4% and template-level F1-score by 59.1% over the simple prompt.The implicit-CoT prompt asked ChatGPT to generate justifications for its decisions.Source: §4.2.3 · p. 9
• On BGL, in-context prompting improved performance over the simple prompt with a few labeled logs, while F1-score and precision peaked at 20 logs and then decreased.Recall was mostly at least 0.99, except at 40 logs, where it was 0.847.Source: §4.2.3 · p. 9, Figure 6 · p. 9

4.2.4 RQ4: Is LogPrompt Compatible With Open-source and Smaller-scale LLMs?

• With LogPrompt, 13B-parameter Vicuna achieved parsing performance comparable to 175B GPT on HDFS, Linux, and Proxifier, and its F1-score rose by 380.7% on average versus a simple prompt.The result supports compatibility with a smaller open-source model.Source: §4.2.4 · p. 10, Table 8 · p. 10

5 DISCUSSION

• 0.450 F1-score for ChatGPT without in-domain training leaves online anomaly detection unsuitable for direct industrial application.The discussion links the difficulty to varied anomaly definitions across systems, which make detection from log text alone challenging.Source: §5 · p. 10
• Interpretation requests were associated with an additional improvement in analysis accuracy, which the authors attribute to an implicit CoT prompting effect.They propose that domain-related CoT samples in pretraining may yield greater improvements.Source: §5 · p. 10
• Practitioners’ positive feedback on generated explanations indicates potential for improved user experience in practical log analysis.The authors suggest that requiring interpretations may also improve accuracy through an implicit CoT effect.Source: §5 · p. 10
• 0.797 F1-score for the self-prompt in log parsing contrasts with 0.450 F1-score as the best result for anomaly detection, motivating task-matched prompt complexity.The authors consider CoT unnecessary for the relatively easier parsing task and more suitable for reasoning-intensive tasks such as anomaly detection [39].Source: §5 · p. 10

6 THREATS TO VALIDITY

• Errors in manually labeled public datasets may lower absolute performance, and the authors cite apparently normal BGL logs labeled abnormal as a possible contributor to LogPrompt’s lower recall there.They note that the raw-log volume makes it infeasible to identify and correct all such errors.Source: §6 · p. 10
• LLM pretraining may have exposed models to log fragments, so stored domain knowledge cannot be ruled out as a factor in LogPrompt’s results.The authors argue that appropriate prompt strategies are needed to elicit such knowledge for log analysis.Source: §6 · p. 10
• Training baselines on limited in-domain data to emulate online use may underestimate their performance and raise fairness concerns, especially against extensively pretrained LLMs.Most baselines are not designed specifically for online scenarios and may require substantially more training data to reach optimal performance.Source: §6 · p. 10
• The human evaluation’s limited log sample prevents statistical significance for usefulness and readability scores, and evaluator qualifications remain uncertain.The practitioners had at least 10 years of software O&M experience, and their positive feedback still indicated promising merits for LogPrompt.Source: §6 · pp. 10–11

7.1.1 Log Parsing.

• Traditional log parsing groups lexically similar logs using clustering, heuristics, or tree-structure mining.Log parsing extracts templates from raw logs to reduce log volume and support downstream analysis.Source: §7.1.1 · p. 11
• Variable-focused parsers recognize variables directly from raw logs, model instance- and message-level semantics, or classify variables into eight categories.This work treats variables as useful for deeper log understanding rather than as material to discard.Source: §7.1.1 · p. 11
• LogPPT uses prompt tuning with a BERT-like model, but its fine-tuning and domain-specific virtual-prompt prediction constrain adaptability and interpretive capabilities.Source: §7.1.1 · p. 11

7.1.2 Log Anomaly Detection.

• Anomaly detection methods either classify using normal and abnormal training logs or forecast from normal historical logs; examples include LogRobust, DeepLog, and LogAnomaly.Other cited classifiers use CNNs or BERT on raw logs.Source: §7.1.2 · p. 11

7.2 LLMs in Software O&M

• LogPrompt addresses log parsing and anomaly detection with three advanced prompt strategies, evaluates interpretability, and requires no in-domain training.The paper contrasts this scope with prior LLM work focused on log parsing and simpler prompts.Source: §7.2 · p. 11
• Earlier LLM studies fine-tuned models for incident recommendations or adapted them using system configuration files, unlike LogPrompt’s general prompt strategies.The paper says these approaches used large amounts of domain-specific training data.Source: §7.2 · p. 11

8 CONCLUSION

• LogPrompt’s primary and unique advantage is furnishing justifications alongside predictions to facilitate program comprehension.Practitioners anticipated uses including unfamiliar-domain event comprehension, analysis-report composition, and ruling out false alarms; the approach is also compatible with open-source and smaller-scale LLMs.Source: §8 · p. 11
• Future work includes domain adaptation, validation on more LLMs, and extending prompt strategies to additional tasks.Source: §8 · p. 11