PaperlayerView exact revision on arXiv

Source-linked AI summary

SLS-Bench: A Benchmark for Incident Log Summarization with Synthetic Observability Data

Valerii Iakovlev, Markus Spanring, Martin Flechl

arXiv:2610.16412v1cs.AI

TL;DR

Evaluating whether agents can produce comprehensive incident summaries from large log streams remains an open question, and collecting real-world benchmark data is costly. SLS-BENCH grounds synthetic tasks in real-world incident reports and evaluates LLMs, finding frontier models useful but costly and slow, while open-weight models lag.

  • Problem

    Whether current agents can produce comprehensive, high-quality summaries from incident logs remains open, while collecting diverse production logs and expert reference summaries at scale is prohibitively expensive.

    Source: §1 · pp. 1–2

  • Method

    SLS-BENCH uses real-world incident reports and a structured incident model to guide synthetic log-stream and reference-summary generation, with validation and statistical comparisons to real log streams.

    Source: §1 · p. 2, §3.1 · p. 4

  • Results

    Evaluations of 14 LLMs find frontier models produce strong summary quality at significant cost and latency, while open-weight models lag substantially in quality.

    Source: §1 · p. 2

  • Takeaways & Limitations

    SLS-BENCH supports systematic evaluation and improvement of LLM agents on incident summarization, and its pipeline offers a scalable blueprint for synthetic observability datasets.

    Source: §5 · p. 10

    Logs and reference summaries come from the same incident model, and judges score against references without seeing logs, so evaluation depends on references faithfully representing the logs’ evidence.

    Source: §5 · p. 10

Abstract

from arXiv · show

As IT systems grow in complexity, automated agent-driven incident response becomes increasingly critical. A key capability for these agents is summarizing large log streams into human-readable reports. Further progress requires meaningfully evaluating this function, yet existing benchmarks leave end-to-end summarization from large log streams underexplored due to the prohibitive cost and complexity of collecting real-world data. We introduce SLS-Bench, a benchmark for incident log summarization featuring 90 tasks grounded in real-world incident reports, each with a synthetic log stream and a reference summary. To construct SLS-Bench, we developed a data generation pipeline that converts incident reports into structured incident models, which guide the generation of synthetic logs and reference summaries. This pipeline combines narrative grounding, explicit incident modeling, generator-verifier loops, and statistical comparisons with real-world log streams to produce realistic and controllable observability data. Our evaluation of 14 LLMs on SLS-Bench shows that while frontier models can produce useful summaries, they do so at substantial cost and latency, and open-weight models lag significantly. SLS-Bench provides a framework for systematic evaluation of log analysis agents and a scalable blueprint for future synthetic observability benchmarks.

1 Introduction

Evaluating end-to-end incident-log summarization remains an open challenge: existing benchmarks do not cover the practical conditions operators face, and collecting real-world logs and expert summaries at scale is costly. SLS-BENCH addresses this gap with a report-grounded synthetic benchmark and pipeline, then evaluates 14 LLMs.

1 Introduction

  • Current agents’ ability to analyze incident logs and produce comprehensive, high-quality summaries remains an open question, while existing benchmarks do not cover large log volumes, diverse systems and failures, realistic scenarios, and comprehensive structured summaries together.Prior benchmarks simplify at least one dimension, such as using short snippets or keyword-style outputs.Source: §1 · p. 1
  • Collecting benchmark data at scale is prohibitively expensive because it requires diverse production systems, observed failures, relevant logs, and substantial expert effort for reference summaries.Source: §1 · p. 2
  • SLS-BENCH uses real-world incident reports to generate synthetic logs and reference summaries, grounding tasks in descriptions of system components, topology, technologies, symptoms, timelines, and root causes.Source: §1 · p. 2
  • Generating the benchmark requires both realistic, detailed log streams from high-level reports and reference summaries that faithfully connect the incident narrative to log-level evidence.Source: §1 · p. 2
  • 90 tasks each pair a large log file averaging 56k logs with a comprehensive reference summary, and the benchmark evaluates 14 LLMs, finding frontier models costly and slow while open-weight models lag in quality.The pipeline uses incident models, generator–verifier loops, verification measures, and statistical comparisons with real-world log streams.Source: §1 · p. 2

2 Related Work

Prior work addresses log analysis through specialized subtasks, LLM-based methods, and agentic diagnostic workflows. Existing benchmarks, however, generally evaluate limited contexts or small, task-specific outputs rather than complete incident summaries from raw log streams.

Task-level log analytics

  • Traditional log parsers, anomaly detectors, and root-cause analysis methods return granular outputs rather than comprehensive human-readable incident summaries [He et al., 2017, Dai et al., 2020, Huo et al., 2021, Liu et al., 2022; Du et al., 2017, Meng et al., 2019, Zhang et al., 2019, Guo et al., 2021; Chen et al., 2014, Ma et al., 2020, Wu et al., 2020, Lin et al., 2024].Source: §2 · p. 3

LLMs for log analysis

  • LLM log-analysis work applies in-context learning, fine-tuning, or stepwise reasoning, but evaluations remain centered on limited contexts and brief outputs [Le and Zhang, 2023, Jiang et al., 2023a, Xu et al., 2024, Wu et al., 2024, Liu et al., 2023, Qi et al., 2023; Huang et al., 2023, Ji et al., 2024, Liu et al., 2024; Ma et al., 2025].Source: §2 · p. 3

Agentic log analysis

  • Agentic log-analysis systems manage multi-step diagnosis, but their evaluations remain localized to components and short explanations rather than complete post-incident reports from raw streams.Source: §2 · p. 3

Log analytics benchmarks

  • Existing benchmarks evaluate tasks such as template prediction, single-entry questions, short-window summaries, top-k root causes, or detection and mitigation, not comprehensive stream-level failure summaries [Jiang et al., 2023b; Huang et al., 2023; Ji et al., 2024; Meng et al., 2023; Cui et al., 2024; Liu et al., 2024; Pham et al., 2025; Xu et al., 2025; Chen et al., 2025; Wang et al., 2026].Source: §2 · p. 3

3 Benchmark

SLS-BENCH construction turns incident reports into structured models that drive synthetic logs and reference summaries, with artifact verification and calibrated LLM judging supporting benchmark evaluation.

3 Benchmark

  • SLS-BENCH comprises 90 tasks seeded by unique incident reports, each pairing a fixed question with a large log file and comprehensive reference summary.The 90 reports were selected from 343 collected reports after de-duplication and filtering; GPT-5.2 achieved 91% filtering precision, with κ = 0.81 across repeated filtering runs.Source: §3 · p. 3, §3.1 · p. 4

3.1 Pipeline Components

  • Incident models provide a structured, controllable representation that guides both log and summary generation, translating report context into topology and emission behavior.Models specify components and their log templates, emission mechanisms, and a phased timeline; the simulator instantiates these specifications as concrete logs, while summaries are constrained to log-observable implications.Source: §3.1 · pp. 4–6

3.2 Pipeline Component Generation

  • A generator–verifier loop checks artifacts against type-specific criteria, combining algorithmic and semantic validation for incident models and simulators.Reference answers receive LLM-based semantic verification; failed checks return feedback for revision, with up to 20 iterations per object before a restart.Source: §3.2 · p. 6
  • K consecutive verifier passes improve pass rates and yield verifier-stable artifacts; verifiers detected 98% of injected requirement violations.At K=1, objects often failed later checks; some failures at K=10 reflected overly rigid rubric interpretations.Source: Figure 4 · p. 6, §3.2 · pp. 6–7
  • SLS-BENCH matches Loghub-2.0 on message length, variables per template, and variable cardinality, but has fewer templates, flatter rank–frequency curves, and lower adjacent-template predictability.The lower template count reflects simulation of the incident-relevant subsystem; the authors report that ambiguity, incomplete evidence, and conflicting signals are harder to formalize and quantify.Source: §3.2 · p. 7

3.3 LLM Judge Scoring

  • The judge scores summaries across seven dimensions using five labels; GPT-5.2 achieved 90% calibration accuracy, while Sonnet 4.6 had 62% independent calibration accuracy.Sonnet was added to counterbalance GPT-5.2's occasional penalties for harmless additions, though it was more prone to missing genuine rubric violations.Source: §3.3 · pp. 7–8
  • Across 14 evaluated LLMs, the two judges' mean scores differed by no more than 0.12, and their final model rankings correlated at ρSpearman = 0.98.Final scores average the judges' predictions after mapping the five rubric labels to continuous values.Source: Figure 6 · p. 7, §3.3 · p. 8

4 Results

Across 90 benchmark problems, frontier models approach the reliable-reconstruction threshold but remain costly and slow, while open-weight models lag; score dimensions and trajectory analysis expose remaining evidence and efficiency challenges.

Evaluation setup

  • The baseline evaluates closed-source frontier and open-weight models using an LLM agent with a stateful Python interpreter and standard packages.The evaluation reports mean scores across 90 problems, plus average monetary cost and runtime per problem.Source: §4 · p. 8

Main results

  • OpenAI models define most of the cost–performance Pareto frontier, while MiniMax-M2 is also Pareto-optimal at low cost.Among closed-source models, performance generally scales with model size and reasoning effort; lower reasoning effort offers the best cost- and time-efficiency ratios within a size class.Source: §4 · pp. 8–9
  • GPT-5.4 medium and high averaged 0.73 against the 0.75 reliable-reconstruction threshold, costing $1.56–$2.19 and 6.9–9.5 minutes per problem.Open-weight models lagged closed-source counterparts, with GLM-5 achieving the strongest open-weight score.Source: §4 · p. 9

Disaggregated score analysis

  • Diagnosis nears the maximum score of 1, but timeline progression and support alignment typically plateau at 0.7–0.8.Low support-alignment predictions often omit or misinterpret critical evidence used by the reference answer.Source: §4 · p. 9, Figure 7 · p. 9

Efficiency analysis

  • In 10 GPT-5.4 medium trajectories, 21% of Python outputs were precomputable, and the initial orientation-and-planning prefix accounted for roughly 30% of tool outputs.After this prefix, agents typically branch into independent analytical streams that could be parallelized.Source: §4 · p. 9

5 Conclusions

SLS-BENCH uses incident reports to generate synthetic logs and reference summaries, and its evaluation finds frontier models viable but costly while open-weight models lag. The conclusion presents the benchmark and pipeline as tools for systematic evaluation and future synthetic observability datasets.

5 Conclusions

  • Figure 8 shows precomputable Python tool outputs and indicates that, after initial orientation, sequential tool calls can be parallelized into independent streams.Source: Figure 8 · p. 9
  • The benchmark contains 90 distinct tasks derived from unique real-world incident reports; evaluation of 14 LLMs finds frontier models viable but costly and slow, while open-weight models lag in summary quality.The pipeline combines real-world grounding, explicit incident modeling, strict validation, and statistical comparisons with real log streams.Source: §5 · p. 9
  • Coverage is limited to 90 publicly available reports, and the incident model caps represented behavior complexity, excluding richer systems and some production complexity.The authors identify expanding report coverage and modeling richer systems, failure modes, background noise, and incidental production complexity as future work.Source: §5 · pp. 9–10
  • Logs and reference summaries come from the same incident model, while judges score against references without seeing logs, making evaluation depend on references faithfully representing log evidence.The authors identify direct transfer evaluation on real incident-log summarization as a next step; judge-robustness tests and a small human study support the current scoring approach.Source: §5 · p. 10
  • SLS-BENCH supports systematic evaluation and improvement of LLM agents on realistic incident summarization, while its pipeline offers a scalable blueprint for synthetic observability datasets.The authors publicly released data and code to support reproducible generation, evaluation, and future extensions.Source: §5 · p. 10

Appendix Overview

  • The appendices cover report collection, incident modeling, statistical analyses, prompts, scoring, generation costs, agent configuration, tool budgets, trajectories, and evaluation robustness.Appendix J includes judge robustness, human evaluation, and comparisons with external benchmarks.Source: Appendix Overview · p. 14

A.1 Collection and preprocessing

  • From 659 candidate URLs, preprocessing yielded 343 reports, deduplicated them to 289, and an LLM filter selected 90.Human inspection preceded adding candidate URLs to the source pool.Source: §A.1 · p. 14

A.2 Filtering

The appendix describes how incident reports are filtered and represented as incident models, and how those models capture outage progression and recovery. It also presents checks for realism, consistency, and observable evidence, along with a distributed-service example and a deterministic simulator that converts scenario events into chronological synthetic logs.

A.2 Filtering

  • Figure 9 summarizes failure types and organizations; checks of the top five organizations found unique systems and failure types, except for one Cloudflare incident retained in two substantially different descriptions.Organization counts below 3 are omitted from the figure.Source: §A.2 · p. 15, Figure 9 · p. 15

A.3 Licensing

  • The benchmark is released under CC BY 4.0 and its code under Apache 2.0, while linked reports remain subject to publishers’ rights and terms.Released artifacts include incident models, simulators, synthetic logs, reference answers, benchmark questions, and metadata; report URLs are provided without redistributing report text, HTML, or images.Source: §A.3 · p. 15

B.1 Terminology and incident-model format

  • Incident models use YAML files for static system architecture and the dynamic incident timeline, with burst, background, and request-path log emissions.The format supports reliable LLM generation and structural verification against a fixed schema.Source: §B.1 · p. 16

B.2 High-level fidelity, consistency, and realism of the log streams

  • High-level realism is pursued by grounding incident models in real reports and requiring ten consecutive verifier passes for faithfulness and internal consistency.Models can represent partial observability and uncertainty, interleaved distributed activity, and the minimal subsystem needed to explain an incident.Source: §B.2 · p. 18

B.3 Concrete incident-model example: 0b9e6bdf

  • The Buildkite example models a hosted CI dashboard that degrades while the Agent API continues polling during the incident.Its 50-minute scenario contains five ordered events; the topology shows the dashboard path and Agent API sharing PostgreSQL.Source: §B.3 · p. 19, Figure 11 · p. 19, Figure 10 · p. 19
  • In the example incident, PostgreSQL saturation triggers failed dashboard health checks and ELB 503s; capacity changes partially restore checks before database, bootstrap, and pooling fixes reduce failures.Operators first lock desired capacity and register borrowed instances; later they upgrade RDS, roll out a fixed AMI bootstrap path, and roll back pooling topology.Source: §B.3 · pp. 21–22

B.4 Incident model statistics

  • Across 90 accepted models, the median had six components, 27 log templates, eight request-flow variants, and four failure events, with maxima of 211 hosts and 180 variable slots.The generation prompts target compact models that remain reliable to generate and verify while capturing realistic system structures and incident progression.Source: §B.4 · p. 22, Table 2 · p. 22

B.5 Simulator example and generated logs

  • A deterministic Python simulator converts scenario events into piecewise-constant failure segments, emits background logs, one-shot events, and request flows, and writes chronologically sorted CSV; incident 0b9e6bdf yields 43368 lines.The CSV contains timestamp, level, message, trace_id, service, and host fields.Source: §B.5 · pp. 22–23

C Log-Statistics Comparison Details

The comparison finds that SLS-BENCH resembles Loghub-2.0 in several template and variable distributions, but its templates are less concentrated and its logs less locally predictable. The appendix also specifies a fixed incident-analysis question and a structured, evidence-grounded report format.

C Log-Statistics Comparison Details

  • The comparison uses Loghub-2.0 files and SLS-BENCH logs, examining message length, template counts, variables, rank–frequency, and adjacent-template predictability.Loghub-2.0 templates use released EventTemplate annotations; SLS-BENCH templates come from incident-model system descriptions.Source: C Log-Statistics Comparison Details · pp. 24–25, Figure 12 · p. 24
  • Message-length distributions overlap substantially, although SLS-BENCH messages are narrower in distribution and longer on average.The authors attribute this partly to more variables and long identifiers such as trace or request IDs.Source: Figure 12 · p. 24, C Log-Statistics Comparison Details · p. 25
  • SLS-BENCH has fewer unique templates than some Loghub-2.0 systems because its incident models include only the minimal relevant system subset.Mac and Linux have 626 and 338 templates, respectively; unrelated templates could be injected to reduce the signal-to-noise ratio.Source: Figure 12 · p. 24, C Log-Statistics Comparison Details · p. 25
  • Variable counts overlap significantly, and variable-cardinality distributions have similar shapes, with most variables taking only a few values.SLS-BENCH has slightly more variables per template on average; only a small fraction of variables have many distinct values.Source: Figure 12 · p. 24, C Log-Statistics Comparison Details · p. 25
  • Real logs have steeper rank–frequency curves and greater adjacent-template predictability, while SLS-BENCH uses templates more broadly and is less locally predictable.Both datasets have heavy-tailed template frequencies, but real streams are dominated by a few templates.Source: Figure 12 · p. 24, C Log-Statistics Comparison Details · p. 25

D.1 Fixed question

  • The fixed benchmark prompt asks the agent to analyze a log file and write an incident investigation report.The example reference answer is shortened in the appendix; released benchmark files contain the full answer.Source: §D.1 · p. 25, D Fixed Question and Example Reference… · p. 25
  • The report instructions partition the log into chronological phases and require summaries, log-surface evidence, and reasoning with calibrated certainty.Each phase is compared primarily with the immediately preceding phase.Source: §D.1 · p. 26
  • The requested report also includes a primary diagnosis and a reconstructed system, with claims tied to visible log evidence and uncertainty stated explicitly.The instructions separate the fault chain from amplifiers, mitigation, recovery, alternatives, and ambiguity.Source: §D.1 · pp. 25–26

D.2 Example reference answer excerpt

The example reference answer traces dashboard failures to database saturation and explains why scaling did not promptly restore service. The judge rubric evaluates both report structure and incident semantics, then combines axis scores into an overall verdict.

D.2 Example reference answer excerpt

  • Partial mitigation produced mixed 200/500 checks and some slow logins, but EC2 API limits later reduced successful launches and left the system degraded.Major RDS, AMI, and PgBouncer changes aligned with recovered metrics and mostly successful checks, though latency and intermittent errors persisted.Source: §D.2 · p. 26
  • PostgreSQL saturation caused DB-dependent dashboard health checks to fail, ELB to drain targets, and widespread 503s; bootstrap failures and missing-shard checks prolonged recovery.The exact cause of the database saturation remains uncertain.Source: §D.2 · p. 27

E LLM Judge Rubric

  • The judge separates report structure from incident semantics across seven evaluation axes and combines their scores into five ordered overall verdicts.Diagnosis and timeline are the main story anchors; support alignment and uncertainty assess whether the account is supported and appropriately qualified.Source: E LLM Judge Rubric · pp. 27–28, Table 3 · p. 28, Table 4 · p. 29

F Dataset Construction Cost

The cost accounting covers token usage and expenses across pipeline stages and their generator and verifier steps. Verification costs around twice as much as generation, while feedback-driven regeneration is the main generation expense.

F Dataset Construction Cost

  • Table 5 breaks down token usage and cost by pipeline stage and separates generator from verifier usage and expenses.Cached tokens are a subset of input tokens, and reasoning tokens are a subset of output tokens.Source: Table 5 · p. 30, F Dataset Construction Cost · p. 30
  • Verification is around twice as expensive as generation.The comparison describes the reported cost asymmetry across those pipeline steps.Source: F Dataset Construction Cost · p. 30
  • Feedback iterations account for most generation cost because each update requires the generator to regenerate the object; in-place editing could reduce that expense.The authors identify more efficient feedback handling as a way to lower generation costs and widen the cost asymmetry.Source: F Dataset Construction Cost · p. 30

G Evaluation Details

The evaluated log-analysis agent uses a ReAct-style workflow with a stateful Python interpreter to investigate logs. The setup specifies model access, sampling parameters, a fixed user prompt, and tool-use instructions.

G Evaluation Details

  • The agent iteratively reasons about actions and issues tool calls through a stateful Python interpreter with pandas and standard Python packages available.Persistent interpreter state lets it investigate logs without rewriting code for every iteration.Source: G Evaluation Details · p. 30
  • OpenAI models were accessed through Azure, while Anthropic and open-weight models used Bedrock; all used default temperature and top-p settings.The setup allowed up to 200 tool calls per problem and a 500k output-token limit per request.Source: G Evaluation Details · p. 30
  • The user prompt follows the question template, and the system prompt directs the agent to answer questions about logs using the provided CSV and Python notebook.The instructions also limit direct printing of large objects and request only the final answer.Source: G Evaluation Details · pp. 30–31

H Tool-call-budget sensitivity

Across the tested call budgets, scores generally improve as budgets grow, while model ordering stays the same at every nonzero budget. The analysis also reports how budgets relate to call use and tool-output size.

H Tool-call-budget sensitivity

  • The study evaluates seven model configurations on 30 sampled problems using budgets of 0, 5, 10, 25, and 50 calls.The agent receives the remaining budget with each tool output, and outputs are capped at 12,000 characters and 1,000 lines.Source: H Tool-call-budget sensitivity · p. 31
  • All models score zero without tool calls; scores generally improve with larger budgets, and model ordering is preserved across nonzero budgets.Table 6 reports means for a random sample of 30 problems.Source: Table 6 · p. 31, H Tool-call-budget sensitivity · p. 31
  • With 50 calls available, DeepSeek V3.2 and GPT-5.4 Nano (medium) average about 37 calls, while Haiku 4.5 and Opus 4.6 (low) average about 16 and 11.The authors describe these call counts as indicating budget under-utilization.Source: Table 7 · p. 31, H Tool-call-budget sensitivity · p. 31
  • At a budget of 50, average tool-output lengths are 0.49–0.91 times the five-call-budget values.Per-call output size generally decreases as the budget grows; the authors say this is consistent with more concentrated inspection when calls are scarce.Source: Table 8 · p. 32, H Tool-call-budget sensitivity · p. 32

I Efficiency Analysis of Agent Trajectories

The trajectory analysis finds substantial orientation work and identifies generic artifacts that could be precomputed, followed by independent analyses that could run concurrently.

I Efficiency Analysis of Agent Trajectories

  • 75 of 361 outputs (21%) are precomputable generic log artifacts available before the agent commits to an incident hypothesis.These include schema, scale, inventories, severity distributions, representative messages, summaries, and coarse temporal views.Source: I Efficiency Analysis of Agent… · p. 32
  • 108 of 361 outputs (29.9%) fall in the orientation-and-planning prefix, which ends when downstream evidence substantiation begins.The counts come from ten GPT-5.4 (medium) trajectories, with Python tool outputs as the unit of analysis.Source: I Efficiency Analysis of Agent… · p. 32
  • After orientation, analyses of timelines, templates, traces, support-plane events, and recovery checks can fan out independently.A parallel implementation could run these analyses concurrently and aggregate their results for report synthesis.Source: I Efficiency Analysis of Agent… · p. 32
  • Once shared orientation is fixed, four stream families can run concurrently and be aggregated before report synthesis without changing the serialized trajectory's evidence.The dependency graph is fan-out/fan-in: the streams do not depend on one another's intermediate outputs.Source: I Efficiency Analysis of Agent… · p. 33

J.1 Judge robustness

The judge-robustness tests compare scores for summaries that preserve or alter reference meaning despite different levels of lexical overlap. Judges score meaning-preserving rephrasings highly and penalize meaning changes.

J.1 Judge robustness

  • Opus 4.6 rephrased all 90 reference summaries, while meaning-altering modifications aimed to retain structure and lexical content; ten examples of each were manually checked.The rephrasings were instructed to preserve meaning and change wording and structure; the altered versions changed meaning and substance.Source: §J.1 · p. 34
  • The experiments suggest that judges assess summary meaning and substance rather than relying on superficial structure or lexical matching.Scores remained near-maximal under rephrasing and strongly penalized substantive changes.Source: §J.1 · p. 34
  • 0.99 average score for meaning-preserving rephrasings versus 0.17 for meaning-altering changes, with respective word-Jaccard similarities of 0.56 and 0.87.Table 11 reports the two judges' mean scores and their average, used for SLS-Bench scoring.Source: Table 11 · p. 34, §J.1 · p. 34

J.2 Human evaluation

In a five-problem human evaluation, human and SLS-BENCH scores produced identical model rankings. The study was limited in scale and used a condensed rubric and access to incident models.

J.2 Human evaluation

  • The study sampled five models spanning the full-benchmark score range and five SLS-BENCH problems, yielding 25 predicted summaries.The same sampled problems were used for every model; the evaluator was blinded to model identities.Source: §J.2 · p. 34
  • The comparison is limited to one evaluator and five problems, and the condensed rubric limits direct comparison of absolute scores.The evaluator also received the incident model as an overview of the log stream's structure.Source: §J.2 · p. 34
  • The combined SLS-BENCH scores and human scores produce identical model rankings over five shared problems.Human scores were higher for the strongest models, consistent with the condensed rubric's greater leniency.Source: §J.2 · p. 34, Table 12 · p. 35

J.3 Comparison with external benchmarks

This section discusses SLS-Bench rankings in relation to o11y-bench and OpenRCA, alongside LLM use in data generation and answer scoring. The ranking comparison is directional because the benchmarks use different tasks and grading criteria.

J.3 Comparison with external benchmarks

  • o11y-bench evaluates observability and SRE workflows in a live Grafana stack, while OpenRCA scores root-cause localization from production telemetry.The o11y-bench comparison uses 15 model configurations; the OpenRCA comparison uses its 51-problem Telemetry subset and five models spanning SLS-BENCH scores.Source: §J.3 · p. 35, Table 14 · p. 36
  • 0.81 overall Spearman rank correlation with o11y-bench; correlations were 0.83 for OpenAI, 1.00 for Anthropic, and 0.90 for open-weight models.Anthropic’s ordering was preserved exactly, while GPT-5.4 Nano (high) ranked above both GPT-5.4 Mini configurations on o11y-bench.Source: §J.3 · p. 35, Table 13 · p. 35
  • 0.90 Spearman rank correlation between SLS-BENCH and OpenRCA Partial scores, despite a ranking reversal involving Haiku 4.5 and GPT-5.4 Nano (low).Haiku 4.5 ranked above GPT-5.4 Nano (low) on SLS-BENCH but below it on both OpenRCA metrics.Source: §J.3 · p. 36, Table 14 · p. 36
  • Strong rank agreement with two independent real-world benchmarks provides evidence that SLS-BENCH performance predicts agent performance on real data and different log analytics tasks.The authors treat these as directional comparisons because the benchmarks differ in tasks and grading criteria, and OpenRCA covers only five models across 51 cases.Source: §J.3 · p. 36
  • LLMs serve as generators and verifiers in synthetic-data generation and as judges of predictions against ground-truth answers; the authors open-source the code and prompts.The paper says further usage details appear in Section 3 and the Appendix.Source: NeurIPS Paper Checklist · p. 43
Loading 2610.16412v1…