PaperlayerView exact revision on arXiv

Source-linked AI summary

Benchmarking Small Language Models and Small Reasoning Language Models on System Log Severity Classification

Yahya Masri, Emily Ma, Zifu Wang, Joseph Rogers, Chaowei Yang

arXiv:2601.07790v1cs.AI

TL;DR

Manual interpretation of massive, complex system-log streams is infeasible, while existing benchmarks provide limited evidence about small models’ comprehension of real-world Linux operational logs. The paper benchmarks SLMs and SRLMs on journalctl severity classification under zero-shot, few-shot, and RAG prompting, measuring accuracy and latency. Results show strong model and retrieval stratification, with Qwen3-4B reaching 95.64% accuracy with RAG and several reasoning models degrading under retrieval, while the benchmark targets real-time digital-twin deployment.

  • Problem

    Existing LM log-analysis research lacks a comprehensive benchmark for real-world operational Linux logs and remains fragmented across log types, tasks, and modeling paradigms.

  • Method

    The study evaluates SLMs and SRLMs on journalctl severity classification under zero-shot, few-shot, and RAG prompting, measuring accuracy and inference latency.

  • Results

    Qwen3-4B achieves 95.64% accuracy with RAG, while retrieval substantially improves some SLMs but degrades several SRLMs.

  • Takeaways & Limitations

    Severity classification functions as a probe of runtime log comprehension and real-time deployability for compact models in digital-twin monitoring and diagnostic contexts.

  • Takeaways & Limitations

    The benchmark’s prior-work gap is scoped to real-world Linux operational logs, prompting strategies, retrieval behavior, and deployable SLM/SRLM severity classification.

Abstract

from arXiv · show

System logs are crucial for monitoring and diagnosing modern computing infrastructure, but their scale and complexity require reliable and efficient automated interpretation. Since severity levels are predefined metadata in system log messages, having a model merely classify them offers limited standalone practical value, revealing little about its underlying ability to interpret system logs. We argue that severity classification is more informative when treated as a benchmark for probing runtime log comprehension rather than as an end task. Using real-world journalctl data from Linux production servers, we evaluate nine small language models (SLMs) and small reasoning language models (SRLMs) under zero-shot, few-shot, and retrieval-augmented generation (RAG) prompting. The results reveal strong stratification. Qwen3-4B achieves the highest accuracy at 95.64% with RAG, while Gemma3-1B improves from 20.25% under few-shot prompting to 85.28% with RAG. Notably, the tiny Qwen3-0.6B reaches 88.12% accuracy despite weak performance without retrieval. In contrast, several SRLMs, including Qwen3-1.7B and DeepSeek-R1-Distill-Qwen-1.5B, degrade substantially when paired with RAG. Efficiency measurements further separate models: most Gemma and Llama variants complete inference in under 1.2 seconds per log, whereas Phi-4-Mini-Reasoning exceeds 228 seconds per log while achieving <10% accuracy. These findings suggest that (1) architectural design, (2) training objectives, and (3) the ability to integrate retrieved context under strict output constraints jointly determine performance. By emphasizing small, deployable models, this benchmark aligns with real-time requirements of digital twin (DT) systems and shows that severity classification serves as a lens for evaluating model competence and real-time deployability, with implications for root cause analysis (RCA) and broader DT integration.

1 Introduction

System logs support monitoring, diagnosis, and automated responses, but their scale and contextual complexity make manual interpretation infeasible. The study therefore uses severity classification as a controlled probe of compact models’ ability to ground operational log semantics under deployment constraints.

  • System logs record operational events across distributed systems and support fault diagnosis, health monitoring, automated responses, and root cause analysis.
  • Massive log volumes and context-dependent language exceed human review capacity, delaying fault detection and increasing mean time to resolve.
  • RAG supplies relevant external evidence at inference time, improving grounding for domain-aware and knowledge-intensive tasks.
  • Syslog severity combines facility and severity values into a priority used to identify errors, but inconsistent labeling makes it a challenging probe rather than canonical ground truth.
  • The study evaluates SLMs and SRLMs on journalctl logs as a probe of runtime log comprehension under constrained outputs and latency requirements.

2 Related work

Prior log-analysis research spans traditional machine learning, deep learning, and language models, but remains fragmented across datasets, tasks, and domains. This work addresses the resulting gap with a journalctl severity-classification benchmark that jointly evaluates prompting, retrieval, accuracy, and latency for deployable SLMs and SRLMs.

  • Prior ML and DL methods: Traditional and deep learning methods have achieved strong results on benchmark log datasets, but may miss temporal dependencies or incur computational and generalization costs.
  • LM log-analysis gap: Current LM research lacks a comprehensive benchmark for noisy, irregular production system logs, often relying instead on code, security, telecom, or structured anomaly datasets.
  • LM log-analysis gap: Prior prompting and RAG studies emphasize intrusion detection, anomaly detection, parsing, or summarization rather than severity classification on operational Linux logs.
  • LM log-analysis gap: Existing efficiency studies do not evaluate inference latency or real-time severity classification performance on journalctl data.
  • Research objectives: The benchmark evaluates SLMs and SRLMs on real-world journalctl severity classification across zero-shot, few-shot, and RAG prompting.
  • Research objectives: Accuracy and inference latency are jointly analyzed to assess deployability in real-time digital-twin and monitoring pipelines.

3 Data

The benchmark is built from millions of journalctl entries collected across six servers, standardized with operational metadata, and sampled into a deduplicated, stratified evaluation set. Its design preserves rare high-severity events while retaining representative lower-severity coverage for model assessment.

  • Collection and preprocessing: Over 7.3 million log entries from six servers were extracted from system journals, converted to structured JSON, and tagged by source IP.
  • Collection and preprocessing: The logs span June 2024 to July 2025 and retain timestamps, process IDs, commands, messages, and priority levels.
  • Sampling and balancing: 46,774 unique logs remained after sampling 50,000 entries, retaining all priority-1–4 logs and balancing lower-severity levels 5–7.
  • Dataset split: An 80/20 stratified split produced 37,419 training logs and 9,355 evaluation logs, with labels removed before inference to prevent leakage.
  • Severity distribution: The standardized syslog scale covers levels 0–7 from emergency to debug, but no emergency-level entries were observed.
  • Benchmark purpose: The resulting semi-balanced benchmark is intended to represent operational conditions while gauging model comprehension relevant to digital-twin integration.

4 Methodologies

The study benchmarks small language and reasoning models for Syslog severity classification using three prompting strategies, real-world server logs, and accuracy and latency evaluation. RAG adds dynamically retrieved training examples to zero-shot prompting, while model families and architectures span efficient SLMs and reasoning-oriented SRLMs.

  • Dataset and evaluation workflow: 7.3 million log entries from six servers were standardized into structured JSON and sampled to form a semi-balanced benchmark.The benchmark applies zero-shot, few-shot, and RAG prompting across multiple open-source models.
  • Prompting strategies: Zero-shot prompting asks models to classify logs without examples or prior context using Linux system-log specialist role framing.The prompt emphasizes system administration, log analysis, journalctl syntax, and Syslog standards.
  • Prompting strategies: Few-shot prompting adds five training-subset examples covering routine informational messages through critical disk errors.The examples remain unseen during evaluation and are intended to expose linguistic and structural severity cues.
  • Prompting strategies: RAG indexes 37,419 labeled training logs as 768-dimensional embeddings and retrieves semantically similar examples to augment the zero-shot prompt.FAISS uses L2 similarity with a default retrieval depth of k=5; retrieved snippets are appended as in-context examples.
  • Models and evaluation: The evaluation compares SLMs and SRLMs across accuracy and per-log inference latency using a reproducible LM Studio deployment on dual RTX A6000 GPUs.The model groups include Llama3.2 and Gemma3 SLM variants alongside Qwen3 and DeepSeek-R1-Distill-Qwen SRLMs.

5 Results

Results show that prompting strategy strongly affects severity-classification accuracy, with RAG producing the largest gains for many models but substantial failures for some reasoning models. Accuracy improvements are accompanied by important output-following and latency trade-offs.

  • Zero-shot: 33.61% was the highest zero-shot accuracy, achieved by Qwen3-1.7B, while smaller Llama3 and Gemma variants remained below 10%.Zero-shot inference time also varied widely, from 0.08 seconds per log for Llama3.2-1B to several seconds for some reasoning models.
  • Few-shot: 56.01% was the best few-shot accuracy, achieved by Qwen3-4B, while Llama3.2-1B and Phi-4-Mini-Reasoning fell to 0% after producing verbose, noncompliant outputs.Few-shot prompting improved most architectures but increased latency, reaching 39.33 and 23.43 seconds per log for these two models.
  • RAG: 95.64% was the highest RAG accuracy, achieved by Qwen3-4B; Gemma3-1B reached 85.28% and Qwen3-0.6B reached 88.12%.RAG also raised latency, with Qwen3-4B requiring 7.14 seconds per log.
  • RAG: RAG performance declined sharply for Qwen3-1.7B, DeepSeek-R1-Distill-Qwen-1.5B, and Phi-4-Mini-Reasoning.The reported drops were 43.30% to 28.96%, 17.63% to 3.17%, and 0% accuracy for Phi-4-Mini-Reasoning, respectively.
  • Overall observations: RAG benefits were therefore substantial but non-universal, requiring model- and size-specific evaluation under operational constraints.The results identify architecture and model scale as important considerations when selecting a domain-specific severity-classification system.

6 Discussion

Across prompting strategies, models separate clearly in both accuracy and efficiency: RAG usually helps, but retrieval integration fails for some models, while latency varies widely. The discussion relates these differences to retrieval depth, architecture, post-training, and output constraints.

  • Retrieval depth: 28.96% at k=5 declined to 26.47% at k=1 for Qwen3-1.7B, even as inference speed improved.Reducing retrieval depth did not recover accuracy, indicating that excessive context volume was not the main problem.
  • Overall observations: 95.64% with RAG made Qwen3-4B the top performer, while Gemma3-1B rose from 20.25% few-shot accuracy to 85.28% with RAG.Qwen3-0.6B also rose to 88.12%, whereas Qwen3-1.7B and DeepSeek-R1-Distill-Qwen-1.5B degraded under RAG.
  • Efficiency: 11 of 27 evaluations completed inference in under one second, while Gemma variants generally stayed below 1.2 seconds per log.Latency therefore remains a separate deployment axis from classification accuracy.
  • Architectural factors: Qwen3-0.6B’s strong RAG result is associated with strong-to-weak distillation, whereas Qwen3-4B combines broader capacity with reasoning-oriented post-training.The discussion presents these architectural and training characteristics as possible explanations for their retrieval performance.
  • Architectural factors: Gemma3 models exploit retrieved examples through interleaved local and global attention blocks despite having fewer query heads than Qwen3.This provides a distinct architectural account of their effective use of retrieved context.
  • Architectural factors: Several SRLMs may over-weight internal reasoning over retrieved snippets when strict single-digit outputs are required.The paper frames this as a plausible explanation for verbose or malformed outputs, high latency, and RAG-related accuracy drops.

7 Conclusion and Future Work

The study finds that retrieval effects vary across small models: Qwen3-4B benefits strongly from RAG, while Qwen3-1.7B degrades, suggesting architecture and training shape external-context integration. These results should be interpreted as evidence about retrieval behavior and context utilization, not comprehensive measures of log understanding or diagnostic competence.

  • Model-specific retrieval behavior: Retrieval effectiveness depends on architecture, training objectives, and interactions between external context and internal reasoning dynamics.Several SRLMs degrade under RAG even though retrieval consistently strengthens SLM performance.
  • Scope and interpretation: Severity-label accuracy and latency indicate retrieval integration behavior and context utilization efficiency rather than direct correctness in RCA or anomaly diagnosis.Severity levels are noisy, administrator-defined, and weakly standardized across systems, limiting their suitability as ground truth for comprehensive log understanding.
  • Model-specific retrieval behavior: 56.01% →95.64%: Qwen3-4B shows substantial RAG improvement, whereas Qwen3-1.7B degrades and Qwen3-0.6B relies heavily on near-label exemplars.The authors associate these differences with model capacity, stability, and how reasoning-oriented models integrate retrieved evidence.
  • Scope and interpretation: The mixed SRLM outcomes suggest a potential tension between generative reasoning and retrieval, but the study does not make a definitive claim about SRLM behavior.Prior work on long-context limitations, inconsistent chain-of-thought, and retrieval-aware training provides plausible explanations and mechanisms.
Loading 2601.07790v1…