Source-linked AI summary
Towards Understanding LLM-Based Log Anomaly Detection: An Empirical Study of Performance, Efficiency, and Robustness
Bin Li, Dongdong Wang, Siyang Lu
TL;DR
Accuracy-focused evaluations offer only a partial view of practical trade-offs in LLM-based log anomaly detection. The paper evaluates adaptation strategies, architectures, scales, efficiency, quantization, and noise across three public datasets, finding substantial performance differences across strategies, dataset-dependent scaling gains, different computational costs among comparable-accuracy models, and largely preserved accuracy under low-bit quantization.
Problem
Predictive metrics such as Precision, Recall, and F1-score provide only a partial view of practical performance because inference time, adaptation overhead, deployment footprint, quantization, and noisy logs also matter.
Source: §1 · p. 1Method
The paper applies a six-dimensional evaluation framework to benchmark adaptation strategies, model architectures, parameter scales, and quantization settings across BGL, HDFS, and Thunderbird.
Source: §1 · p. 1Results
Across three public datasets, adaptation strategies show substantial performance differences, scaling gains vary by dataset, comparable-accuracy models have different computational characteristics, and low-bit quantization largely preserves detection accuracy in evaluated configurations.
Source: §5 · p. 4Takeaways & Limitations
Model selection should consider efficiency and robustness alongside detection accuracy under practical deployment conditions.
Source: §5 · p. 4
Abstract
from arXiv · showhide
Large language models (LLMs) have demonstrated promising performance in log anomaly detection, yet how their adaptation strategies, architectures, and deployment configurations affect detection effectiveness remains insufficiently understood. To investigate these factors, we conduct a systematic empirical analysis across three public log datasets, examining different adaptation strategies, model architectures, parameter scales, and quantization settings. Our results reveal substantial performance differences across adaptation strategies, while model scaling yields varying detection gains across datasets. We further observe that models with comparable detection accuracy can exhibit markedly different computational costs, and that low-bit quantization largely preserves detection performance in the evaluated configurations. Finally, we examine detection robustness under structural, semantic, and label noise at different perturbation levels. These findings provide empirical insights into the performance, efficiency, and robustness of LLM-based log anomaly detection, highlighting practical considerations beyond conventional accuracy-oriented evaluation.
1. INTRODUCTION
The introduction argues that accuracy-focused evaluations miss practical costs and reliability concerns, and frames a six-dimensional empirical evaluation across three public log datasets.
1. INTRODUCTION
- Predictive metrics provide only a partial view because higher accuracy may require more inference time, adaptation overhead, or deployment capacity, while quantization and noisy logs may affect reliability.The introduction argues that accuracy alone cannot fully characterize practical trade-offs.Source: §1 · p. 1
- The proposed framework evaluates detection performance, inference efficiency, adaptation efficiency, deployment efficiency, quantization resilience, and noise robustness across BGL, HDFS, and Thunderbird.It benchmarks representative adaptation strategies, model architectures, model scales, and quantization settings.Source: §1 · p. 1, Figure 1 · p. 2
- Across three public datasets, adaptation strategies and model architectures show performance differences, while detection gains from model scaling vary across datasets.The analysis investigates adaptation strategies, architectures, and parameter scales.Source: §1 · p. 1
- Comparable detection accuracy can entail different computational costs across inference, adaptation, and deployment.The analysis examines trade-offs among detection performance, inference efficiency, adaptation overhead, and deployment cost.Source: §1 · p. 1
- Low-bit quantization largely preserves detection performance, while sensitivity varies under structural, semantic, and label noise.These robustness effects are evaluated as part of the broader analysis.Source: §1 · p. 1
2. EXPERIMENTAL DESIGN
The experimental design evaluates LLM-based log anomaly detection across three public datasets and varies adaptation, architecture, scale, and quantization. It measures detection performance alongside efficiency and robustness, including resilience to quantization and injected noise.
2.1. Datasets and Evaluation Protocol
- Experiments use BGL, HDFS, and Thunderbird, preserving chronological order so earlier sequences support adaptation and later sequences are used for evaluation.This sampling reflects deployment on future events after adapting on historical logs and reduces information leakage between training and test sets.Source: §2.1 · p. 2
2.2. Evaluated Models and Adaptation Strategies
- The evaluated configurations vary adaptation strategy, model architecture, parameter scale, and quantization precision.The design includes white-box and prompt-based approaches, encoder-only, encoder–decoder, and decoder-only models, three GPT-2 sizes, and Llama2-7B at 16-, 8-, and 4-bit precision.Source: §2.2 · p. 2
2.3. Evaluation Metrics
- Detection is evaluated using Precision, Recall, and F1-score, with F1-score as the primary metric and dataset-specific results reported alongside the average.The average is computed across the evaluated datasets.Source: §2.3 · p. 2
- Efficiency measures cover inference latency, adaptation overhead, and deployment model size, with higher normalized inference, adaptation, and deployment scores indicating greater relative efficiency.Adaptation efficiency uses trainable versus total parameters; deployment efficiency uses a model-size proxy.Source: §2.3 · pp. 2–3
- Quantization resilience measures F1 preservation after precision reduction, while noise robustness measures F1 preservation under structural, semantic, and label noise at injection ratios of 0.05, 0.15, and 0.25.The quantization and noise scores are averaged across their corresponding evaluation settings; higher scores indicate greater preservation of detection performance.Source: §2.3 · p. 3
3. DETECTION PERFORMANCE ANALYSIS
Detection performance varies substantially with adaptation strategy, while architecture comparisons are dataset-dependent and do not isolate architecture alone. Increasing model scale can improve detection, but the gains vary across datasets.
Impact of adaptation strategies
- Frozen-layer tuning and LoRA achieve average F1-scores of 99.08 and 98.89, while Prompt-Tuning achieves 98.83; hard-prompt methods score 57.56–60.93.The reported hard-prompt scores are Zero-shot 57.56, Few-shot 59.83, and CoT 60.93; strategy has a substantial impact on detection performance.Source: §3 · p. 3
Impact of model architectures
- Architecture averages range from 95.76 for T5 to 99.17 for DeepSeek-7B, but scores cluster on Thunderbird and range from 92.10 to 99.60 on HDFS.The evaluated models also differ in parameter scale and pretraining characteristics, so the comparisons do not isolate architecture alone.Source: §3 · p. 4
Impact of model scale
- GPT-2 Medium, Large, and XL average F1-scores are 57.04, 91.90, and 94.77, respectively, with scaling gains varying substantially across datasets.On BGL the variants score 97.62–98.35; on Thunderbird, Medium scores 10.26 versus 97.86 and 98.00 for Large and XL.Source: §3 · p. 4
4. EFFICIENCY AND ROBUSTNESS ANALYSIS
Models with similar detection performance can differ in computational characteristics, while low-bit quantization largely preserves detection performance in the evaluated configurations. The noise analysis finds greater tolerance to structural perturbations than to semantic or label noise.
Detection performance vs. computational efficiency
- DeBERTa, LoRA, and DeepSeek-7B achieve average F1-scores of 98.29, 98.89, and 99.17, respectively, while exhibiting different inference, adaptation, and deployment efficiency scores.GPT-2 XL achieves an average F1-score of 94.77, compared with 99.17 for DeepSeek-7B, while their computational characteristics differ.Source: §4 · p. 4
- Practical assessment considers detection performance alongside inference latency, trainable parameters, and model size.The analysis describes computational characteristics as complementary to detection metrics when considering practical trade-offs.Source: §4 · p. 4
Impact of model quantization
- 98.89, 99.04, and 98.98 are the average F1-scores for Llama2-7B at 16-bit, 8-bit, and 4-bit precision, respectively.Across BGL, HDFS, and Thunderbird, F1-scores range from 99.58 to 99.87, 98.94 to 99.30, and 98.03 to 98.16, respectively; deployment benefits require separate computational-cost assessment.Source: §4 · p. 4
Robustness under noisy logs
- Structural noise causes less degradation than semantic and label noise, particularly at higher injection ratios, for LoRA-tuned Llama2-7B.The evaluation uses 5%, 15%, and 25% injection ratios; Fig. 2 reports detection performance across these noise settings.Source: §4 · p. 4, Figure 2 · p. 4
5. CONCLUSION
Across three public datasets, the paper examines LLM-based log anomaly detection and finds that performance differences, efficiency trade-offs, and robustness patterns vary across evaluated configurations. It concludes that evaluation should consider detection performance, computational efficiency, and robustness together.
5. CONCLUSION
- The study analyzes LLM-based log anomaly detection across three public datasets, finding substantial performance differences among adaptation strategies and dataset-varying gains associated with model architectures and parameter scales.Source: §5 · p. 4
- Models with comparable detection performance can exhibit different computational characteristics, while low-bit quantization largely preserves detection accuracy in the evaluated configurations.Source: §5 · p. 4
- The paper examines a LoRA-tuned detector's sensitivity to structural, semantic, and label noise and concludes that evaluation should consider performance, efficiency, and robustness together.Source: §5 · p. 4