PaperlayerView exact revision on arXiv

Source-linked AI summary

SRE-Marathon: A Continuous, Change-Driven Benchmark for Autonomous Site Reliability Agents

Yifang Tian, Yingjian Bai, Yifeng He, Zichun Chong, Yuanchen Gao, Yiran Li, Hans-Arno Jacobsen

arXiv:2609.33023v1cs.AIcs.SE

TL;DR

Production SRE agents must handle noisy alerts, overlapping incidents, and faults arising from code or configuration changes, unlike the isolated tasks used by typical benchmarks. SRE-Marathon evaluates sustained operation in a live system with seeded faults and periodic agent access. Across ten methods, the best Marathon-Score is 41.3 out of 100, and agents rarely complete repairs while faults remain active.

  • Problem

    Typical SRE-agent benchmarks use isolated incidents, while production operations involve noisy alerts, overlapping problems, and faults that can originate in code or configuration changes.

    Source: §1 · p. 1

  • Method

    SRE-Marathon evaluates agents operating a live microservice deployment under continuously injected, seeded faults, with periodic invocations and offline scoring from recorded evidence.

    Source: §1 · p. 2, §7 · p. 9

  • Results

    41.3 out of 100 is the best Marathon-Score across ten methods and two backbones; agents often correlate and localize faults but rarely complete timely repairs as load increases.

    Source: §7 · p. 10

  • Takeaways & Limitations

    SRE-Marathon exposes a gap between agents’ ability to diagnose faults and their ability to turn diagnosis into repair during sustained operation.

    Source: §7 · p. 10

    The short-run comparison does not isolate load experimentally because venue, invocation interval, and seed also differ.

    Source: §5.1 · p. 8

Abstract

from arXiv · show

Benchmarks for site reliability engineering (SRE) agents are typically episodic: one fault is injected, the agent receives an incident task, and its response is scored. Production operation is not. Incidents surface through noisy alerts, overlap in time, and often originate from code or configuration changes. We present SRE-Marathon, a benchmark for long-horizon, continuous SRE operation. An agent is invoked at a fixed cadence with cumulative alert history and a persistent workspace while operating a live two-zone Kubernetes deployment as a fault orchestrator injects overlapping faults according to a seeded, production-calibrated schedule. Curated code and configuration changes deployed through the same build pipeline available for repair. Each run is recorded into a sealed bundle and scored offline: Marathon-Score credits each injected fault for ordered progress through correlation, localization, and repair, with all metrics computed deterministically from recorded system evidence. Across three applications and about sixty faults per run, the best of 10 methods reaches only 41.3 out of 100. Agents often correlate and localize faults, but almost never complete repairs while the faults remain active.

1 INTRODUCTION

SRE-Marathon addresses the gap between episodic incident benchmarks and production operations, where noisy alerts, overlapping incidents, and faults from system changes complicate diagnosis and repair. It evaluates sustained operation across the full incident lifecycle; across ten methods, the best Marathon-Score is 41.3 out of 100.

1 INTRODUCTION

  • Production operation requires engineers to interpret noisy signals, determine which symptoms belong together, identify changes, and choose remedies while incidents may overlap.Code and configuration bugs together account for 39.5% of high-severity incidents in a study of a large cloud service.Source: §1 · p. 1
  • Existing benchmarks typically inject one fault into a clean environment, give an explicit incident task, and stop after the agent submits an answer.Their episodic setup leaves noisy-signal triage, internal code or configuration failures, overlapping incidents, collateral effects, and operational cost largely unmeasured.Source: §1 · p. 2
  • SRE-Marathon evaluates sustained production operations with continuous faults, evolving alerts, live microservices, and realistic code and configuration changes.Its evaluation spans detection and correlation, root-cause localization, repair, and recovery validation.Source: §1 · p. 2
  • 41.3 out of 100 is the best Marathon-Score across ten methods, while agents often correlate faults and some localize root causes but successful repair remains rare.The benchmark records runs into sealed bundles and computes metrics deterministically offline from system evidence.Source: §1 · p. 2
  • SRE-Marathon’s contributions include an offline metric suite for correlation, localization, repair, intervention risk, service degradation, latency, and cost.The metrics are computed deterministically from recorded system evidence.Source: §1 · p. 2
  • The benchmark evaluates ten methods spanning runbooks, SRE agents, and coding agents on two model backbones.Its empirical evaluation includes broad performance comparisons and failure analysis.Source: §1 · p. 2

2 THE SRE-MARATHON DESIGN

SRE-Marathon places an agent in a live microservice deployment and periodically asks it to infer and manage incidents from operational signals. A seeded orchestrator injects reproducible, potentially overlapping faults, while instrumented tools expose observations and controlled interventions.

2 THE SRE-MARATHON DESIGN

  • Each run combines a live application under traffic, a seeded fault sequence with hidden ground truth, and an agent interface whose interactions are recorded.Hidden ground truth includes the responsible service and artifact; it is retained by the benchmark but not exposed to the agent.Source: §2 · p. 3
  • The agent operates a live application, so evaluation can measure recovery, user-facing degradation, and collateral effects after interventions.The benchmark includes Astronomy Shop, Online Boutique, and Train Ticket; Astronomy Shop additionally uses two Kubernetes zones with replicated instances and failover.Source: §2 · p. 3
  • The seeded orchestrator makes the injected fault sequence reproducible for the same benchmark release and random seed.It determines which faults occur, where and when they begin, and when reverted faults are removed.Source: §2 · p. 3
  • The fault catalog spans three applications, and each fault targets one service while allowing symptoms to propagate through application dependencies.The catalog records the failure mechanism, target, affected scope, symptom probe, and responsible artifact.Source: §2 · pp. 3–4, Table 1 · p. 4
  • Change-driven faults use plausible code or configuration changes, with code defects deployed through the normal release pipeline after passing lint and unit tests.Direct faults instead use controlled deployment interventions such as resource exhaustion or network disruption.Source: §2 · p. 4
  • Faults arrive continuously under an increasing schedule, remain active for four to twelve minutes, and may overlap as operational load accumulates.The agent receives no fault-start notification and must detect problems through their observable effects.Source: §2 · p. 4
  • The interface exposes operational signals and controlled actions through instrumented tools, while a policy layer constrains and records mutating operations.Read-only access includes alerts, metrics, logs, traces, topology, Kubernetes state, and source code; interventions include rollback, configuration changes, scaling, and source repair.Source: §2 · pp. 4–5
  • At each fixed interval, the agent receives cumulative alert history and a persistent workspace, but each invocation runs in a fresh process.This gives every agent a fixed number of opportunities to act regardless of alert volume; long-run invocations use a 120-second interval and a 110-second budget.Source: §2 · p. 5

3 EVALUATION

The evaluation scores each injected fault through ordered correlation, localization, and repair stages using recorded system evidence. It also reports safety, service-impact, timing, and cost outcomes.

The Marathon-Score

  • Marathon-Score tracks each fault through ordered correlation, localization, and repair, using recorded system evidence and including every injected fault in its denominator.Localization contributes only after correlation, and repair only after both earlier stages; attempted forbidden operations also affect the score.Source: §3 · p. 5

Incident-handling stages

  • Correlation matches faults to reported incidents containing attributable alerts, while localization requires identifying the exact culprit service or services and responsible artifact.Alerts can be compatible with multiple simultaneously active faults; this occurs for 39%, 76%, and 47% of alerts in the three applications.Source: §3 · p. 5
  • Repair counts an agent intervention only when system evidence confirms recovery before the orchestrator reverts the fault.Recovery requires a symptom check to return to and remain healthy, or verification that the responsible artifact was corrected when no symptom failure was observed.Source: §3 · pp. 5–6

Operational outcomes

  • Intervention risk is the fraction of actions that exceed an incident’s affected scope or violate an explicit safety policy.Forbidden attempts include deleting the benchmark namespace or accessing protected secrets.Source: §3 · p. 6
  • Service degradation measures cumulative user-facing errors or excessive latency weighted by entry-point traffic, relative to a matched no-action run.The degraded-request fraction combines failed requests and requests slower than 2.5 seconds, capped at one.Source: §3 · p. 6
  • The evaluation reports median detection, localization, and repair times over faults reaching each stage, plus model calls, token use, and estimated monetary cost.Timing metrics use the agent invocation interval as their resolution, and their values need not be ordered because they use different fault subsets.Source: §3 · p. 6

4 EXPERIMENTAL SETUP

The experiments use a seeded marathon schedule across three microservice applications and compare ten methods under shared alerts, tools, and policy enforcement. Runs use a reproducible single-host cluster, with an additional seven-node deployment validation.

Schedules and deployment environments

  • Each run injects around sixty faults during 60 minutes of increasing intensity, then observes for 30 minutes without new injections, with agents invoked every 120 seconds.Average simultaneous faults are 5.8, 6.1, and 6.5 across Online Boutique, Train Ticket, and Astronomy Shop; peak concurrency is 10, 11, and 11.Source: §4 · p. 6
  • The main experiments use a single-host Kind cluster for reproducible comparison and additionally validate the benchmark on a seven-node kubeadm deployment.All methods share seeded schedules for each of the three applications.Source: §4 · p. 6

Methods

  • Ten methods are compared, including rule-based, tool-augmented, retrieval, persistent-state, commercial-agent, coding-agent, and diagnosis approaches under shared alerts, tools, and policy enforcement.LLM-based methods use Kimi K2.5 or GPT-5.6 Luna with a budget below $8 per run.Source: §4 · p. 6
  • Table 2 averages each method’s marathon runs across three microservice applications and marks the best value per column.The table is the SRE-Marathon leaderboard.Source: Table 2 · p. 7

5 RESULTS

Across methods, diagnosis is more common than successful repair, and the gap widens under sustained operation. Case studies show that even a correct diagnosis may not lead to timely remediation.

5.1 OVERALL BENCHMARK PERFORMANCE

  • 41.3 is Keep-agent’s highest Marathon-Score, but only Keep-agent completes any fault end to end; repair remains at most 0.024 while faults are active.Keep-agent handles 1/57 faults on Online Boutique with Kimi K2.5 and 2/59 with GPT-5.6 Luna; most score comes from correlation and localization.Source: §5.1 · p. 7
  • Shorter ReAct runs score higher than marathon runs, but venue, invocation interval, and seed differences prevent isolating load experimentally.The passage describes the comparison as consistent with increasing concurrency being a major source of failure, not as an isolated causal test.Source: §5.1 · pp. 7–8

5.2 PERFORMANCE ANALYSIS UNDER SUSTAINED OPERATION

  • As load increases, tool calls rise mainly through incident reporting, while actions remain sparse and stage-wise repair stays weakest.ReAct reports grow from 3.9 to 35.5 per invocation; no Kimi K2.5 method repairs more than 0.06 of injected faults at any point.Source: §5.2 · p. 8

5.3 CASE STUDIES

  • The same bad-image-tag fault is repaired end to end by Keep-agent on Online Boutique, but Claude Code’s late rollback on Astronomy Shop misses the fault window.Claude Code first makes two ineffective configuration changes and a rejected manifest patch before reaching the correct rollback after three attempts.Source: §5.3 · p. 9

6 RELATED WORK

Prior SRE benchmarks assess diagnosis, fixed incidents, or live-cluster work one episode at a time. SRE-Marathon extends evaluation to continuous live operation with overlapping faults, build-pipeline defects, and deterministic scoring.

Benchmarks for agents

  • SRE-Marathon extends long-horizon agent evaluation to continuous live SRE operation with overlapping faults and deterministic scoring from sealed run recordings.The benchmark also includes defects delivered through the build pipeline.Source: §6 · p. 9
  • Prior SRE benchmarks span frozen-telemetry diagnosis, mocked or recorded incidents, and live-cluster interaction limited to one fault episode at a time.The cited examples distinguish evaluation settings by the telemetry and system access agents receive.Source: §6 · p. 9

LLM agents for the incident lifecycle

  • SRE agents commonly target alert correlation, root-cause analysis, or runbook mitigation, while some systems use multi-agent pipelines across detection and repair.Deployed examples also include coding agents, open-source agents and alert platforms, and provider offerings.Source: §6 · p. 9

7 CONCLUSION

The conclusion presents SRE-Marathon as a sustained-operation benchmark whose recorded evidence enables reproducible offline scoring. The supplied appendix passages detail its environments, fault catalog, validation, alert rules, tools, and worked metric examples.

7 CONCLUSION

  • The conclusion frames SRE-Marathon around sustained production operation and reports a best Marathon-Score of 41.3 out of 100, with repair rarely following diagnosis.Metrics are computed deterministically from recorded system evidence for reproducible offline evaluation.Source: §7 · pp. 9–10

A ENVIRONMENT SETUP

  • The single-host venue uses a 16-vCPU, 32-GB virtual machine hosting a five-node Kind cluster split across two zones, with 25-ms one-way latency injected between zones.The multi-node venue instead uses seven virtual machines for its kubeadm cluster, with no inter-zone latency injection.Source: A ENVIRONMENT SETUP · p. 14

B THE FAULT CATALOG

  • Faults are categorized by failure cause rather than injection mechanism, so a wrong setting remains a configuration fault even when delivered as source code.The catalog contains 700 faults across categories and applications.Source: B THE FAULT CATALOG · p. 14

C GENERATING CODE AND CONFIGURATION FAULTS

  • Code and configuration faults are hand-authored and validated on live deployments to ensure observable failures while requests continue to be served.Validation checks for new messages or other screening criteria, confirms deployment of the change, and tests recovery; a fault validated alone may still be masked by concurrent faults or appear later.Source: C GENERATING CODE AND CONFIGURATION… · pp. 14–15

D THE ALERT RULE PACK

  • The frozen alert pack contains 44 rules built from 18 templates, with five rules applying only to Astronomy Shop.Request rules read the application entry-point service, and “holds for” specifies how long a condition must persist before an alert fires.Source: D THE ALERT RULE PACK · p. 15

E INSTRUMENTED TOOL DETAILS

  • The 19 published tools include free read operations, scored incident reporting, and interventions that count toward risk; guarded calls must cite a reported incident.Forbidden operations include reading secrets, deleting the benchmark namespace, and editing benchmark rules.Source: E INSTRUMENTED TOOL DETAILS · pp. 15–16

F WORKED EXAMPLES FOR EVERY SCORED FORMULA

  • Worked examples show that correlation, localization, and repair are scored per fault, while MTTD, MTTL, and MTTR are medians over faults reaching each stage.The examples also calculate service degradation against a matched no-action run and report model-call, token, and dollar costs.Source: F WORKED EXAMPLES FOR EVERY SCORED… · pp. 16–17

G BASELINES AND CODING AGENTS

The baselines use scripted rules, fresh conversations, retrieval, or persistent incident grouping; ORBIT adds durable cross-invocation state and a bounded decision loop. Codex and Claude Code provide coding-agent baselines under the benchmark’s shared tools and limits.

Runbook

  • Runbook groups alerts by workload and applies a fixed rule-to-action table, reporting the table’s fault category as root cause.When multiple rules fire, a rule naming the broken object takes priority over a symptom rule.Source: G BASELINES AND CODING AGENTS · p. 17, Table 6 · p. 18

ReAct

  • ReAct starts a fresh conversation each invocation, retaining only cumulative alert history and previously opened incidents while choosing its own tool calls.It stops when it stops calling tools, reaches ten steps, or spends 150k tokens.Source: G BASELINES AND CODING AGENTS · pp. 17–18

RAG

  • RAG retrieves live telemetry and topology, then returns a plan that groups alerts, assigns root causes, and may propose repairs for execution.It makes one model call per invocation and ranks retrieved documents against alert names and labels.Source: G BASELINES AND CODING AGENTS · p. 18

Keep-agent

  • Keep-agent combines rule-based alert grouping in a persistent incident store with RAG’s telemetry retrieval and model-generated diagnosis and repair plans.Its rules assign each alert to at most one incident, while the model may place only alerts the rules left ungrouped.Source: G BASELINES AND CODING AGENTS · p. 18

ORBIT

  • ORBIT reloads workspace state across invocations, merges alerts deterministically, and uses a bounded model loop to organize and act on its work.It publishes incident mappings before model consultation and promotes repair recipes to memory only after validated recovery.Source: G BASELINES AND CODING AGENTS · pp. 18–19

Coding agents

  • Codex resumes one conversation from the workspace across invocations, while Claude Code starts fresh sessions and stores agent-written notes in a workspace directory.Both use the shared adapter and tools; Claude Code has restricted file tools and no shell or web access.Source: G BASELINES AND CODING AGENTS · p. 19

H HOW FAR A SINGLE RUN GENERALIZES

ReAct’s results vary substantially across three seeds on Astronomy Shop. The section uses this variation to motivate shared schedules for method comparisons and seeded rollouts for reinforcement learning.

H HOW FAR A SINGLE RUN GENERALIZES

  • ReAct’s Marathon-Score ranges from 7.3 to 20.1 across three Astronomy Shop seeds, while stage shares vary by factors of two to four and Intervention risk by more than four.The authors attribute the spread to the differing fault schedules drawn by each seed.Source: H HOW FAR A SINGLE RUN GENERALIZES · p. 19
  • The benchmark compares methods under one shared schedule because different seeds can produce widely different schedules and results.Source: H HOW FAR A SINGLE RUN GENERALIZES · p. 19
  • The authors identify each new seed as a reinforcement-learning rollout under a different setting.Source: H HOW FAR A SINGLE RUN GENERALIZES · p. 19
  • Table 7 reports ReAct on Astronomy Shop under the shared Table 2 seed and two additional seeds, with one marathon run per seed.Source: Table 7 · p. 20

I AGENT BEHAVIOR UNDER LIGHTER PRESSURE

ReAct repairs more faults in short runs than in marathon runs, while alert correlation changes less across regimes. The contrast is qualitative because the run conditions differ in several ways besides load.

I AGENT BEHAVIOR UNDER LIGHTER PRESSURE

J WHERE THE LIFECYCLE BREAKS

The lifecycle breakdown shows that agents often reach the service of a live fault but rarely repair it, while other methods stall earlier at correlation or localization. Similar aggregate scores can therefore conceal different failure stages.

J WHERE THE LIFECYCLE BREAKS

  • More than half of most methods’ actions hit a live fault’s service, yet many such faults remain unrepaired; 44 of ReAct’s 46 actions and 28 of OpenSRE’s 30 land on faults never repaired.Runbook makes the most actions (179), but only four reach faults later repaired; the passage also notes that service-based matching misses some repairs credited through actions on other affected services.Source: J WHERE THE LIFECYCLE BREAKS · p. 20
  • Figure 8 counts executed actions per invocation and sorts executed actions by target, distinguishing inactivity from actions directed at live faulted services.Its live-fault match is by service and does not establish whether the remedy fit.Source: J WHERE THE LIFECYCLE BREAKS · p. 20, Figure 8 · p. 21
  • Figure 9 assigns each injected fault one first-matching outcome: handled, repaired but not handled, acted on without repair, located without action, correlated only, or missed.Source: Figure 9 · p. 21
  • On Kimi K2.5, methods stall at distinct stages: several most often miss faults, ReAct and ORBIT stop mainly at correlation, and Keep-agent often locates faults without acting.For example, the most common outcomes are misses for RAG, OpenSRE, Codex, and Claude Code (43%, 67%, 64%, and 56%), and correlation for ReAct (47%).Source: J WHERE THE LIFECYCLE BREAKS · pp. 21–22

K LIMITATIONS

The benchmark’s results are bounded by compromises in its environment, fault sampling and attribution, alerting, scoring, and evaluated methods.

K LIMITATIONS

  • The two zones share one host, so zone failure emulates redundancy semantics rather than physical zone independence; public applications also leave memorization risk despite mitigations.Newly written code faults and exact-commit artifact targets reduce, but do not remove, that risk.Source: K LIMITATIONS · p. 22
  • The fault mix follows one provider’s high-severity incidents from 2020–21, and affected scope is measured per fault, making attribution conservative under interactions.Source: K LIMITATIONS · p. 22
  • The frozen alert rules can miss faults below every threshold, and agents cannot add rules to extend monitoring coverage.Source: K LIMITATIONS · p. 22
  • Located requires an exact alias-table match, Repaired treats rollback and corrected patches alike, and degradation uses one no-action recording per release.Source: K LIMITATIONS · p. 22
  • Evaluation uses two budget-friendly, non-frontier LLM backbones and ten methods, which do not cover the field; a purpose-built method may do better.Source: K LIMITATIONS · p. 22

Related papers

Loading 2609.33023v1…