PaperlayerView exact revision on arXiv

Source-linked AI summary

TriCalRAG: A Three-Strategy, Retrieval-Augmented Benchmark for On-Premise LLM-Based Root Cause Analysis in AIOps

Rohit Patel, Susil Kumar Mohanty, Jeenal Chaudhary

arXiv:2609.14762v2cs.DCcs.AIcs.CRcs.ETcs.LG

TL;DR

Operational log analysis needs to support private, resource-conscious incident diagnosis, while aggregate detection scores can conceal prediction imbalance. TriCalRAG benchmarks local LLM incident analysis across datasets and prompting strategies; RAG improves detection and reduces extreme prediction imbalance in the reported benchmark.

  • Problem

    Operational logs make timely incident diagnosis difficult, while cloud-hosted LLMs raise privacy, compliance, latency, and cost concerns for sensitive logs.

    Source: §1 · pp. 1–2

  • Method

    TriCalRAG evaluates structured incident analysis with open-weight LLMs served locally across four log datasets and three prompting strategies.

    Source: §1 · p. 2, §3.1 · p. 6

  • Results

    RAG consistently improves detection and reduces extreme prediction imbalance; seven of eight RAG configurations meet the predicted-positive-rate range, versus two of eight zero-shot configurations.

    Source: §8 · p. 17

  • Takeaways & Limitations

    A moderate F1 score can coexist with near-universal anomaly predictions, so detection performance should be read alongside prediction balance.

    Source: §8 · p. 17

    The study scores only binary anomaly decisions and does not quantitatively evaluate the correctness, usefulness, or faithfulness of generated root-cause and remediation text.

    Source: §7 · p. 17

Abstract

from arXiv · show

Operational logs create a need for private, resource-efficient incident analysis, but aggregate detection scores can conceal severe prediction bias. We present TriCalRAG, a reproducible benchmark for log-anomaly detection with generated root-cause and remediation outputs across BGL, HDFS, Thunderbird, and OpenStack. The primary evaluation compares Qwen2.5-14B and Mistral-Small-22B, served through vLLM on one NVIDIA RTX PRO 6000 GPU (96 GB), under zero-shot, few-shot, and retrieval-augmented generation (RAG) prompting across three data-sampling seeds. We report F1, bootstrap confidence intervals, predicted-positive rates, throughput, and memory use, with DeepLog as a held-out classical baseline. Mistral-Small attains a higher macro-averaged F1 than Qwen2.5-14B (0.644 versus 0.560), whereas Qwen provides approximately twice the throughput. A separate log-probability evaluation compares raw decisions with Contextual Calibration (CC) and Batch Calibration (BC): neither correction consistently improves prediction-balance diagnostics across prompting strategies. Supplementary single-run comparisons extend evaluation to 4-bit Llama-3.1-70B via local Ollama and Claude Haiku 4.5 via Anthropic's cloud API; RAG improves F1 on all four datasets for both models. Claude's reported aggregate F1 increases from 0.566 to 0.695, while the estimated API cost rises from \$0.94 to \$2.15 per 1,000 incidents. Local deployment ablations show approximately 41-fold throughput scaling with batching and 20% lower latency with 4-bit quantization on the tested workload. These findings support retrieval as useful context for anomaly decisions, while the supplementary protocols, unvalidated explanation quality, and prediction-balance diagnostics limit broader claims about RCA accuracy and probabilistic calibration.

1 Introduction

Operational logs make timely incident diagnosis difficult, while cloud-based LLM analysis raises privacy, compliance, latency, and cost concerns. TriCalRAG proposes a reproducible single-GPU benchmark that evaluates detection alongside prediction-balance and deployment behavior.

1 Introduction

  • LLM-based incident analysis can generate root-cause explanations and remediation suggestions in addition to binary anomaly decisions.This contrasts with conventional anomaly detectors that typically produce only an anomalous/normal decision.Source: §1 · p. 1
  • Aggregate detection metrics can obscure prediction imbalance, while cloud APIs raise privacy, compliance, latency, and per-query cost concerns for operational logs.A model predicting anomaly for nearly every incident may achieve high recall and moderate F1 while offering limited discrimination.Source: §1 · p. 2
  • TriCalRAG evaluates open-weight LLMs locally on one 96 GB GPU across four public log datasets and three prompting strategies, with supplementary model checks and deployment measurements.The benchmark includes a DeepLog baseline and measures throughput, memory use, quantization, and API cost.Source: §1 · p. 2
  • The benchmark’s contributions include repeated evaluation across three data-sampling seeds with bootstrap 95% confidence intervals and analysis of prediction balance and RAG.The work also describes a reproducible evaluation harness, deployment measurements, and calibration and generalization studies.Source: §1 · p. 2

2 Related Work

Prior work addresses log detection, LLM-based diagnosis, retrieval, serving efficiency, broad benchmarking, and calibration, often as separate concerns. TriCalRAG brings these dimensions together in a controlled benchmark for single-GPU on-premise incident analysis.

2 Related Work

  • TriCalRAG targets a unified evaluation of detection effectiveness, prediction balance, and deployment efficiency for on-premise LLM-based incident analysis.The related-work review identifies these dimensions as a gap across otherwise distinct research areas.Source: §2 · p. 3, §2.6 · p. 5

2.1 Log Anomaly Detection

  • DeepLog predicts the next log key with an LSTM, while LogAnomaly adds template embeddings; such detectors generally do not generate natural-language causes or remediation.LogHub supplies real-world system-log datasets, including the four used in TriCalRAG.Source: §2.1 · p. 3

2.2 LLMs for Log Analysis and RCA

  • LLM log-analysis work spans structured-output anomaly detection [27], fine-tuned or instruction-tuned models [9] [23], and semantic enrichment; prior RCA studies include production incidents and cloud incidents [1] [35].TriCalRAG instead focuses on single-GPU on-premise feasibility and interpreting detection when aggregate metrics may hide prediction imbalance.Source: §2.2 · pp. 3–4

2.3 Retrieval-Augmented Log Analysis

  • TriCalRAG retrieves complete labeled incidents rather than log templates, treats RAG as one of three controlled prompting strategies, and measures prediction balance and deployment behavior.Related retrieval work includes LogRAG and EagerLog.Source: §2.3 · p. 4

2.4 LLM Serving Systems

  • TriCalRAG measures behavior under a fixed serving stack on one workstation GPU rather than optimizing the serving engine itself.The review describes vLLM [18] and related serving-efficiency techniques, including FlashAttention [5] and LoRA [13].Source: §2.4 · p. 4

2.5 General-Purpose LLM Benchmarking

  • Broader benchmarks such as MMLU [12], HELM [21], and SWE-bench evaluate multiple dimensions; TriCalRAG reports detection metrics alongside prediction-balance and deployment diagnostics.The paper applies multidimensional evaluation to AIOps.Source: §2.5 · pp. 4–5

2.6 Calibration of In-Context Learning

  • TriCalRAG tests Contextual Calibration [36] and Batch Calibration [38] as mechanism checks for prediction imbalance, not as alternative definitions of its predicted-positive-rate diagnostic.The calibration methods are evaluated using a separate next-token log-probability experiment.Source: §2.6 · p. 5

3 TriCalRAG Benchmark Design

TriCalRAG benchmarks structured incident analysis using common incident windows and controlled prompting strategies. Its design covers four public datasets, a retrieval pipeline, primary and supplementary systems, and a summarized evaluation protocol.

3 TriCalRAG Benchmark Design

  • TriCalRAG is a benchmark for evaluating LLM-based incident analysis in on-premise AIOps settings, organized around task, datasets, incident construction, prompts, retrieval, systems, and protocol.The overall evaluation architecture is shown in Fig. 1.Source: §3 · p. 5, Figure 1 · p. 5

3.1 Task Definition

  • Each incident supplies raw operational log lines to an LLM, which returns JSON containing an anomaly judgment, severity, root-cause explanation, and remediation suggestion.Only the binary anomaly decision is quantitatively evaluated; generated root-cause and remediation fields are retained but not automatically scored.Source: §3.1 · p. 6

3.2 Datasets

  • The benchmark uses BGL, HDFS, Thunderbird, and OpenStack, constructing 150 incidents per dataset with equal anomalous and normal examples for 600 incidents per run.Thunderbird sampling is limited to its first two million lines because the raw log is approximately 31.7 GB.Source: §3.2 · p. 6

3.3 Incident Construction

  • Each incident uses a five-consecutive-line window, but this can split evidence from an HDFS block because its labeled log entries may be non-contiguous.The common window is retained for protocol consistency despite the mismatch with HDFS block-level labels.Source: §3.3 · p. 6

3.4 Prompting Strategies

  • Zero-shot prompts contain only the task and incident, few-shot prompts add two fixed examples, and RAG adds three similar labeled incidents while holding the task and output schema constant.The design treats added prompt context as the primary variable.Source: §3.4 · p. 6

3.5 Retrieval Pipeline

  • RAG retrieves the three most similar labeled incidents using embeddings and cosine similarity, excluding the query incident itself.The authors caution that query exclusion avoids direct self-retrieval but does not establish the absence of broader retrieval overlap.Source: §3.5 · pp. 6–7

3.6 Systems Evaluated

  • The primary systems are locally served Qwen2.5-14B-Instruct and Mistral-Small-Instruct-2409; DeepLog is a separate held-out classical baseline, and Llama-3.1-70B and Claude Haiku 4.5 are supplementary single-run checks.The supplementary models are served through Ollama and the Anthropic API, respectively.Source: §3.6 · p. 7, Figure 2 · p. 7

3.7 Benchmark Protocol Summary

  • The primary benchmark combines four datasets, two locally served LLMs, three prompting strategies, and three data-sampling seeds.Table 2 summarizes the benchmark construction and evaluation scope.Source: §3.7 · p. 7, Table 2 · p. 7

4 Evaluation Framework

The evaluation framework separates repeated primary experiments from supplementary single-run checks to avoid implying equal statistical confidence. It organizes the study around detection, prediction balance, mechanism and generalization, and deployment efficiency.

4 Evaluation Framework

  • The framework covers research questions, hardware, primary protocols, metrics, statistical methods, supplementary evaluations, deployment ablations, and reproducibility scope.Repeated primary experiments are separated from single-run supplementary experiments because their protocols differ.Source: §4 · p. 7

4.1 Research Questions

  • The four research questions address detection effectiveness, prediction balance, calibration and model generalization, and deployment trade-offs across throughput, latency, memory, quantization, and API cost.The prediction-balance question asks whether RAG reduces extreme predicted-positive rates relative to zero-shot and few-shot prompting.Source: §4.1 · p. 8

4.2 Systems Evaluated

  • The primary protocol evaluates Qwen2.5-14B-Instruct and Mistral-Small-Instruct-2409 with zero-shot, few-shot, and RAG prompting; DeepLog is evaluated separately on a smaller held-out set.The DeepLog results are treated as complementary rather than directly paired with primary LLM runs.Source: §4.2 · p. 8

4.3 Datasets and Evaluation Unit

TriCalRAG evaluates four log datasets using fixed-size incident samples and structured anomaly-analysis prompts. Its primary comparison varies prompting strategy across two locally served models on a single workstation.

4.3 Datasets and Evaluation Unit

  • Each run uses 150 incidents per dataset, balanced between anomalous and normal examples, across BGL, HDFS, Thunderbird, and OpenStack.Each incident is a five-line log window; the requested JSON includes an anomaly decision, severity, root cause, and remediation, but only the binary decision is quantitatively scored.Source: §4.3 · p. 8

4.4 Hardware and Local Serving

  • The local experiments use one NVIDIA RTX PRO 6000 GPU with 96 GB of VRAM, and record throughput and GPU memory alongside predictive metrics.The single-workstation setting tests whether useful log analysis can be performed without a multi-GPU datacenter-class serving environment.Source: §4.4 · p. 8

4.5 Primary Experimental Protocol

  • The primary protocol compares two models across four datasets and three prompt styles, with each configuration evaluated using three independent data-sampling seeds.Zero-shot uses only the task instruction and incident; few-shot prepends two fixed examples; RAG adds the top three semantically similar labeled incidents retrieved through MiniLM/FAISS.Source: §4.5 · p. 9

4.6 Evaluation Metrics

The study measures classification performance and prediction balance, alongside deployment efficiency. Primary F1 estimates use bootstrap confidence intervals across repeated data-sampling seeds.

4.6 Evaluation Metrics

  • Precision, recall, and F1-score measure classification effectiveness, while predicted-positive rate (PPR) tracks the fraction of incidents labeled anomalous.Because the benchmark has a true anomaly prevalence of 0.50, PPR provides a prediction-balance diagnostic rather than a probabilistic-calibration measure.Source: §4.6 · p. 9
  • Configurations with PPR < 0.15 or PPR > 0.85 are flagged as degenerate; proximity to the 0.50 anomaly rate alone does not establish calibration or correctness.Throughput, GPU memory consumption where applicable, and latency for deployment ablations are also reported.Source: §4.6 · p. 9

4.7 Statistical Methodology

  • Primary configurations use three random sampling seeds, and F1 scores are reported with bootstrap 95% confidence intervals from 1,000 resamples.This repeated protocol applies to the primary Qwen2.5-14B and Mistral-Small-22B experiments; supplementary experiments have a different repetition protocol.Source: §4.7 · p. 9

4.8 Calibration-Correction Baselines

A separate scoring experiment tests Contextual Calibration and Batch Calibration against prediction imbalance. Supplementary evaluations examine larger local and cloud-served models, but are reported outside the primary repeated comparison.

4.8 Calibration-Correction Baselines

  • The calibration experiment applies Contextual Calibration (CC) and Batch Calibration (BC) to next-token log-probabilities for binary anomaly labels, distinct from the main benchmark’s parsed JSON decisions.Its degeneracy counts are treated as a mechanism check, not as numerically interchangeable with primary F1 results.Source: §4.8 · pp. 9–10

4.9 Supplementary Model Evaluations

  • Llama-3.1-70B-Instruct is evaluated in 4-bit quantized form through Ollama under zero-shot and RAG prompting as a single-run supplementary check.The check examines whether dataset-dependent prediction imbalance extends to a substantially larger open-weight model.Source: §4.9 · p. 10
  • Claude Haiku 4.5 is evaluated through the Anthropic API on the same 600 incidents under zero-shot and RAG prompting in a single-run supplementary experiment.The evaluation examines prediction imbalance under a cloud-served model and informs the deployment-cost discussion; supplementary results are not members of the primary statistical comparison.Source: §4.9 · p. 10

4.10 Deployment Ablations

Deployment ablations characterize batching and quantization on the local setup, while the reproducibility scope distinguishes the repeated primary benchmark from separately reported experiments.

4.10 Deployment Ablations

  • The deployment ablations compare Qwen2.5-14B batch sizes from 1 to 128 and bfloat16 against AWQ 4-bit quantization on the same workload.Batching characterizes single-GPU throughput under increasing concurrency; quantization is assessed using F1 and latency.Source: §4.10 · p. 10

4.11 Reproducibility Scope

  • The primary benchmark spans 72 dataset-level repeated configurations: two models, four datasets, three prompt styles, and three seeds.At 150 incidents per dataset, this yields 10,800 incident-level responses; the harness separates loading, prompt construction, retrieval, inference, scoring, and figure generation.Source: §4.11 · p. 10
  • Single-run Llama-3.1-70B and Claude Haiku 4.5 experiments, DeepLog, and calibration correction are reported separately because they lack the complete primary three-seed protocol.The deployment ablations characterize deployment behavior rather than changing the primary model-comparison protocol.Source: §4.11 · p. 10, §4.10 · p. 10

5 Experimental Results

Across the primary benchmark, Mistral-Small has higher macro-averaged F1 than Qwen2.5-14B, while prediction balance varies sharply by dataset and prompting strategy. RAG reduces extreme prediction behavior in the primary models and improves F1 across datasets in a supplementary Llama check; batching and quantization improve deployment efficiency on the tested workload.

5.1 RQ1: Detection Effectiveness

  • Mistral-Small achieves higher mean F1 than Qwen2.5-14B (0.644 vs. 0.560) across the four-dataset macro average.The macro average is over four datasets, three seeds, two models, and three prompt styles.Source: §5.1 · p. 10, Table 3 · p. 11

5.1.1 Per-Dataset Breakdown.

  • RAG yields F1 of 0.92–0.94 on BGL and 0.88–0.89 on Thunderbird, while HDFS F1 never exceeds 0.687.The paper attributes HDFS difficulty to five-line windowing that can split block-level anomalies across evaluation windows.Source: §5.1.1 · p. 11

5.2 RQ2: Prediction Balance

  • RAG keeps 7 of 8 configurations within the predefined predicted-positive-rate range, versus 2 of 8 for zero-shot; few-shot under-predicts on HDFS and OpenStack.Zero-shot Mistral-Small reaches PPR 0.993 on BGL, 0.987 on HDFS, and 1.000 on OpenStack; few-shot PPR falls as low as 0.016.Source: Figure 3 · p. 11, §5.2 · p. 12

5.3.1 Calibration-Correction Baselines.

  • Raw log-probability scoring is degenerate in 11 of 24 configurations, while CC and BC do not consistently improve the balance diagnostics.CC lowers zero-shot degeneracy from 5 to 3 and few-shot from 6 to 5, but introduces degeneracy in two RAG configurations; BC yields 12 degenerate cases versus 11 raw.Source: §5.3.1 · p. 12

5.3.2 Generalization to Llama-3.1-70B.

  • In a single-run Llama-3.1-70B check, RAG improves F1 on every dataset, most on OpenStack (0.222 to 0.618).Under zero-shot, BGL and Thunderbird are over-predicted while HDFS and OpenStack are under-predicted; RAG moves their PPRs to 0.600, 0.680, 0.533, and 0.407, respectively.Source: §5.3.2 · p. 12, Table 5 · p. 13, Figure 4 · p. 13

5.4.1 Batch-Size Scaling.

  • 41×: Qwen2.5-14B throughput rises from 48.3 tokens/s at batch size 1 to 1991.0 tokens/s at batch size 128.Batch elapsed time rises from 1.39s to 4.14s.Source: §5.4.1 · p. 13, Table 6 · p. 13

5.4.2 Quantization Impact.

  • 20%: Qwen2.5-14B latency falls from 5.06s to 4.04s with 4-bit quantization on the tested workload.F1 is 0.739 for the quantized model versus 0.730 for bfloat16; the authors do not treat that difference as statistically established.Source: Table 7 · p. 13, §5.4.2 · p. 14

6 Discussion

The discussion argues that F1 alone does not settle the choice between local models, because prediction balance and deployment conditions matter. It also frames DeepLog and Claude comparisons as constrained checks, with differing protocols and cost coverage limiting direct interpretation.

6 Discussion

  • Mistral-Small’s F1 advantage is concentrated where prediction balance is comparatively stable, while zero-shot results can reflect near-constant positive predictions.The discussion identifies cold-start incidents without similar historical precedent to retrieve as a setting where zero-shot class balance may be less reliable, and recommends reporting a balance diagnostic alongside F1.Source: §6 · p. 14

6.1 Comparison to a Classical Baseline

  • DeepLog achieves F1 = 0.698, above Qwen2.5-14B (0.560) and Mistral-Small (0.644), but its smaller held-out set and different protocol prevent a direct superiority claim.DeepLog reports PPR 0.742 and produces only a binary anomaly flag, not a natural-language root-cause explanation.Source: §6.1 · p. 14

6.2 Comparison to a Cloud API

  • Claude Haiku 4.5’s single-run aggregate F1 rises from 0.566 under zero-shot to 0.695 under RAG, with F1 improving on all four datasets.The aggregate prediction-positive rates are 0.518 and 0.392, respectively; per-dataset results show zero-shot threshold failures on three datasets, removed under RAG.Source: §6.2 · p. 15, Table 8 · p. 15, Figure 5 · p. 15
  • Claude’s reported API expenditure rises from $0.94 to $2.15 per 1,000 incidents with RAG, while local infrastructure costs are unmeasured.The authors characterize this as a single-provider, single-model preliminary comparison, not a definitive cost analysis.Source: §6.2 · p. 16, Table 9 · p. 16

7 Limitations

The benchmark’s findings are bounded by dataset-window mismatch, distinct calibration and detection decision paths, a smaller DeepLog evaluation, limited primary model coverage, and unvalidated generated explanations.

7 Limitations

  • Five-line windows may mismatch HDFS block-level labels, whose related log lines can be noncontiguous; block-aware incident construction is a future direction.The authors say the mismatch likely contributes to lower HDFS performance across models and prompting strategies.Source: §7 · p. 16
  • The initial parser misclassified some Mistral-Small outputs as parse failures; extracting the first-to-last brace-delimited text reduced the overall parse-failure rate to 0.2%.The authors report the issue because it materially affected early results and highlights the need to validate structured-output parsers across model families.Source: §7 · p. 17
  • Calibration experiments use next-token log-probabilities, unlike the primary benchmark’s structured-JSON decisions, so their results are not numerically interchangeable.The authors frame calibration as a mechanism check on label-bias correction, not a restatement of the main F1 results.Source: §7 · p. 17
  • DeepLog uses 240 training sequences and 120 held-out incidents, versus 600 incidents per LLM run, so its result is a reference rather than a paired comparison.Its explicit training split gives it a different evaluation protocol; a larger-scale evaluation may produce different results.Source: §7 · p. 17
  • Only two open-weight models receive the repeated three-seed primary evaluation; supplementary model checks use single runs and have lower statistical confidence.Broader repeated evaluation across model sizes and architecture families remains future work.Source: §7 · p. 17
  • Generated root-cause and remediation text is not quantitatively evaluated, so binary-decision F1 does not establish its semantic quality.Assessing correctness, usefulness, or faithfulness would require human annotation or a carefully validated model-based judging protocol.Source: §7 · p. 17

8 Conclusion and Future Work

Across four real datasets, retrieval-augmented prompting improves detection and reduces extreme prediction imbalance, while models trade off F1, balance, and throughput. The authors also report single-GPU deployment results and identify evaluation boundaries and future directions.

8 Conclusion and Future Work

  • Seven of eight RAG configurations remain within the predefined predicted-positive-rate range, compared with two of eight zero-shot configurations.Across four real datasets, two open-weight models, and three prompting strategies, retrieval-augmented prompting consistently improves detection performance and substantially reduces extreme prediction imbalance.Source: §8 · p. 17
  • Mistral-Small achieves macro-averaged F1 of 0.644 versus Qwen2.5-14B’s 0.560, while Qwen exhibits fewer extreme prediction-balance cases and approximately twice the throughput.DeepLog’s F1 = 0.698 is numerically above both LLMs’ macro-averaged F1, but its smaller held-out set and different protocol make the values complementary rather than a direct ranking; LLM explanation quality is not quantitatively evaluated.Source: §8 · pp. 17–18
  • Neither Contextual Calibration nor Batch Calibration consistently reproduces RAG’s reduction in prediction imbalance; supplementary single-run checks show over-prediction on BGL and Thunderbird and under-prediction on HDFS and OpenStack.The authors suggest behavior may partly depend on dataset characteristics, while noting that broader repeated evaluation across model families is needed to confirm this.Source: §8 · p. 18
  • Batching raises throughput approximately 41×, from 48 to 1991 tokens/s between batch sizes 1 and 128, while AWQ 4-bit quantization cuts latency approximately 20% without measurable degradation on the tested workload.The authors describe single-GPU deployment as practical for this evaluated workload, with prompt design and prediction behavior still important operational considerations.Source: §8 · p. 18
  • Future work includes broader three-seed evaluation across architectures and cloud providers, block-aware HDFS windowing, and rigorous evaluation of generated explanation quality.The authors note that explanation assessment requires an appropriate human- or model-based protocol.Source: §8 · p. 18
  • Extensions to GPU telemetry, BMC sensors, boot-trust events, and bare-metal provisioning use synthetic data and are presented as future directions, not validated experimental results..Source: §8 · p. 18
Loading 2609.14762v2…