Source-linked AI summary
A Self-Defining Operator for Distributed Applications
Shihang Li, Thomas Anderson, Ratul Mahajan, Simon Peter, Luke Zettlemoyer
TL;DR
Operators must restore service quickly, yet agents may re-investigate similar faults from scratch. SDO turns incident resolutions into operational memory and retrieves relevant playbooks with telemetry-based detectors; across 11 applications, it resolves recurring faults up to 2.1× faster.
Problem
Method
SDO builds editable operational memory from incident resolutions, using reusable playbooks and lightweight telemetry detectors to surface applicable knowledge.
Source: §1 · pp. 1–2Results
Up to 2.1× faster resolution of recurring faults and 2.5× fewer tokens versus a memoryless agent across 11 microservice applications.
Source: §7 · p. 12Takeaways & Limitations
The results support operating procedure—detectors, reusable playbooks, and architecture summaries—as a useful form of memory for SRE agents.
Source: §7 · p. 12Operational memory changes only through edits during reflection, rather than being retired by disuse.
Source: §3.4 · p. 6
Abstract
from arXiv · showhide
To operate distributed applications with high availability, operators must rapidly diagnose and mitigate any incident. LLM-based agents can help, but unlike experienced human operators they do not improve as they handle more incidents. Each investigation starts from scratch, re-deriving likely causes and re-testing hypotheses even when a similar fault was resolved in the past. We present a self-defining operator (SDO) that improves with time by updating its operational behavior and memory. When SDO resolves an incident, it distills that experience into playbooks, which describe reusable diagnostic instructions for future incidents with similar faults, and detectors, scripts that take application telemetry as input and output relevant playbook(s). SDO continuously runs the detectors and invokes LLMs only when one fires, so its token cost scales with incident count rather than with operational time. Across 11 open-source microservice applications, SDO cuts resolution time by 52% on recurring faults while keeping per-incident cost low. SDO is open source at https://github.com/self-defining-systems/sdo.
1 Introduction
LLM-based agents lack the accumulated operational knowledge that helps human operators resolve recurring incidents faster. SDO learns from resolutions by creating reusable operational memory and uses detectors to retrieve it, reducing MTTR on recurring faults.
1 Introduction
- LLM-based agents repeatedly re-derive causes and retest hypotheses because they do not accumulate operational knowledge across incidents.By contrast, experienced SREs draw on knowledge of application structure and prior failures to resolve recurring faults faster.Source: §1 · p. 1
- SDO turns incident resolutions into editable operational memory so it can increasingly use prior experience instead of reasoning from scratch.Its memory includes reusable playbooks and an architecture summary that grounds them in the application.Source: §1 · pp. 1–2
- Detectors check application telemetry and surface relevant playbooks, so steady-state monitoring uses code and LLM costs are tied to incidents.The agent-authored detectors run continuously; LLMs are invoked when a detector fires.Source: §1 · p. 2
- 52% lower MTTR on recurring faults is reported across 11 open-source applications as SDO accumulates memory.Detector-based retrieval stays accurate and cheap as memory grows, keeping operating costs low.Source: §1 · p. 2
- SDO is implemented as a Kubernetes controller that monitors with agent-authored detectors and commits operational memory to the application repository.The design includes playbooks for reusable repairs, detectors over live cluster state, and an architecture summary.Source: §1 · p. 2
2 Requirements for Autonomous Operators
Autonomous operation requires detecting incidents, learning from them, and retrieving applicable knowledge efficiently. These requirements reflect the diversity and recurrence of production faults, and the limits of agents that begin each incident cold.
2 Requirements for Autonomous Operators
- An autonomous operator must own detection, accumulate knowledge across incidents, and operationalize memory for efficient retrieval.Continuously reasoning with an LLM is prohibitively expensive because faults are rare relative to uptime.Source: §2 · p. 2
- Production faults span code, configuration, dependencies, resources, upgrades, and infrastructure, while symptoms can be delayed, noisy, or surface outside the faulty component.Roughly 60% of high-severity incidents in one study had non-code root causes.Source: §2 · pp. 2–3
- Incidents recur but rarely match on surface symptoms: one fault can combine causes, and superficially similar incidents can have unrelated causes.Prior knowledge is useful only when it applies without replacing investigation.Source: §2 · p. 3
- Human operators reuse recognition, application knowledge, runbooks, and postmortems, while current agents discard context after each incident.Linking a new incident to related past incidents reduces mitigation effort; Google reports a good playbook lowers mean time to repair by roughly 3×.Source: §2 · p. 3
3 SDO Design
SDO is a long-running Kubernetes operator that deploys applications, monitors their health, and learns from resolved incidents. Its design combines human-readable operational memory with code-based detection and LLM-driven response.
3 SDO Design
- SDO’s five memory artifacts are playbooks, detectors, an outcome log, an architecture summary, and a health objective, maintained as durable, human-editable files.The monitor runs detectors to dispatch subagents; after resolution, the responder reflects and updates memory.Source: §3 · pp. 3–4, Figure 1 · p. 3
3.1 Application Deployment
- During deployment, a deployer and health judge iterate until programmatic checks derived from the human’s health objective report the application healthy.The deployer creates deployment artifacts and an architecture summary; health checks become general-purpose safety-net detectors once the application is live.Source: §3.1 · p. 4
3.2 Response to New Incidents
- When an incident occurs, SDO dispatches a responder from health-detector or human signals; without a relevant playbook, it investigates through tested hypotheses.After a successful response, the responder distills the resolution into a playbook and detector, commits them, and the monitor registers the detector on redeploy.Source: §3.2 · pp. 4–5
3.3 Fast Response to Known Incidents
- Detectors run code over cluster state to surface and parameterize relevant playbooks, while playbooks pair diagnostic and repair instructions with reusable scripts.Detector evaluation alone accounts for steady-state cost, so LLM token cost scales with incident count rather than operating time.Source: §3.3 · p. 5
3.4 Refining Memory Over Time
- SDO refines detector-to-playbook mappings from response outcomes and updates architecture summaries when repository changes affect application structure.It marks playbooks and detectors referencing removed or renamed entities for review, and retains quiet memory rather than pruning it by disuse.Source: §3.4 · p. 6
4 Implementation
SDO separates its agent runtime from a Kubernetes controller and stores operational memory in the application repository. It standardizes agent interactions and validates changes before they are committed.
4 Implementation
- SDO separates an approximately 6,000-line Python agent runtime from an approximately 3,000-line Go monitor and detector controller.The generic agent interface supports a Pydantic AI-based harness and a portable adapter for off-the-shelf coding agents.Source: §4 · p. 6
- MCP servers standardize agent interactions, and commit guards reject memory changes that fail checks such as detector compilation or references to unsaved playbooks.The agent invokes CLI checks while editing memory, and the commit path enforces them.Source: §4 · p. 6
- The controller compiles detectors from the repository and evaluates them on schedule against a read-only, typed snapshot of cluster state.Each detector declares the resource kinds it reads and its evaluation interval.Source: §4 · p. 6
- SDO stores playbooks and architecture summaries as Markdown, scripts as files, and detectors as Go source in the application’s Git repository.Memory changes are auditable commits, and redeployment compiles and registers changed detectors.Source: §4 · p. 6
5 Evaluation
The evaluation tests whether SDO improves incident response as memory accumulates, deploys applications reliably and efficiently, transfers memory to recurring and novel faults, and retrieves playbooks accurately and cheaply. It uses 11 open-source microservice applications, with incident-resolution experiments on four and deployment experiments across all 11.
5 Evaluation
- The evaluation extends SREGym fault scenarios through parameterized templates and multi-fault incident composition, replaying the same deterministic stream across configurations.Incident response is evaluated on four applications; deployment from source is evaluated on all 11, with agents using the same model and tools unless stated otherwise.Source: §5 · p. 7
5.1 Incident Resolution Effectiveness
- As detectors and playbooks accumulate, SDO’s rolling time to resolution falls below all baselines, and its token usage also falls.Early in the stream, SDO behaves like the baselines because it has little fault-specific memory; Figures 2 and 3 report timeline trends and distributions, while Table 2 reports solve rates.Source: §5.1 · p. 8, Figure 2 · p. 8, Figure 3 · p. 8
- Detectors reduce search by localizing memory to resources implicated by cluster state, while playbooks turn prior work into reusable procedures and scripts.The agent still checks the live cluster and can reject a misleading playbook when evidence contradicts it.Source: §5.1 · p. 8
5.2 Deployment from Source
- Both deployment configurations successfully deploy all 11 applications; operational memory reduces deployment time and token use by 47.5% and iterations by 2.5×.Figure 4 reports deployment time, iteration count, and token usage; deployments are evaluated using external load generators.Source: §5.2 · p. 9, Figure 4 · p. 10
- The independent health judge prevents premature success declarations: without it, only four deployments passed external validation, compared with all 11 with the judge.The single-agent deployer sometimes declared partial deployments complete; the judge keeps validation anchored to the user-facing load generator.Source: §5.2 · p. 9, Table 3 · p. 9
5.3 Memory Transfer
- On recurring faults, SDO’s median resolution time is 282 s rather than 584 s and token use is 2.2M rather than 5.4M; solve rates are similar to Memoryless.Four of 30 incidents fall on the slower tail when SDO falls back to open-ended diagnosis; on novel faults, resolution time and accuracy remain similar to Memoryless, though detectors surface false positives in 9 of 30 incidents.Source: §5.3 · p. 10, Figure 5 · p. 10, Table 4 · p. 11
- On novel incidents, frozen detectors can surface irrelevant playbooks, but agents inspect and discard them without misdiagnosis or slower resolution.Because test memory is frozen, agents cannot prune mis-firing detectors as they could in live operation.Source: §5.3 · p. 10
5.4 Retrieval Accuracy and Cost
- Detectors retrieve about as accurately as Playbook-LLM and outperform Playbook-Embed on every metric, with no LLM cost at retrieval time.Compared with Playbook-LLM, detectors trade some precision for roughly 45× less time; on no-overlap incidents, Playbook-LLM avoids spurious retrieval best, followed by detectors.Source: §5.4 · p. 11, Table 5 · p. 11
6 Related Work
SDO differs from prior system-operations and self-improvement agents by persisting auditable operational memory, retrieving it from live cluster state, and operating in production. Its playbooks also differ from generic retrieval stores and human-authored runbooks through deployment-specific parameterization and executable scripts.
LLM agents for system operations
- SDO persists each session as auditable memory in agent-authored scripts, directly invocable and retrieved by checking live cluster state rather than symptom similarity, keeping it aligned with continuously changing deployments.Source: §6 · p. 12
Self-improving LLM agents
- SDO accumulates executable artifacts like Voyager [55], rather than text lessons such as Reflexion [49], ExpeL [65], and Agent Workflow Memory [56], while keeping its model and detector API fixed and revising only operational memory.Unlike self-rewriting systems STOP [61], Gödel Agent [58], and the Darwin Gödel Machine [63], SDO's changes remain auditable and reversible; it uses the running cluster as supervisory signal.Source: §6 · p. 12
Memory and long context for LLMs
- Unlike embedding-based memory and retrieval-augmented generation [42] [35], SDO uses a per-application world model and parameterized causal chains with executable scripts, designed to remain useful under deployment drift.The passage says loading raw history is costly, dilutes attention, and lacks the parameterization needed to survive drift.Source: §6 · p. 12
Runbooks and incident management
- SDO's agent-authored playbooks extend practitioner-authored runbooks and procedures with deployment-entity parameterization and executable scripts, and can import human runbooks as seed material.Prior systems recommend existing human-authored guides [64], whereas SDO authors its guidance.Source: §6 · p. 12
7 Conclusion
Across 11 microservice applications, SDO turns incident resolutions into operational memory and improves on a memoryless agent on several reported outcomes. The conclusion identifies detectors, playbooks, and architecture summaries as the components of that memory.
7 Conclusion
- Across 11 microservice applications, SDO reduces deployment time and token use by up to 47.5%, resolves recurring faults up to 2.1× faster, and uses 2.5× fewer tokens than a memoryless agent.The conclusion also reports the highest end-to-end incident solve rate among the compared agents.Source: §7 · p. 12
- The paper concludes that useful SRE-agent memory is operating procedure: detectors retrieve applicable knowledge, playbooks encode reusable repairs, and architecture summaries ground both in the evolving application.SDO improves with experience by turning incident resolutions into operational memory.Source: §7 · p. 12
A Responder Prompt
The responder prompt defines a workflow for diagnosing incidents, restoring verified end-to-end health, and converting findings into reusable operational guidance. It also specifies how to organize maintainable playbooks and detectors under `.sdo/` and link detected symptoms to relevant guidance.
Resolve the Incident
- During investigation, the responder externalizes checks, fixes, hypotheses, and findings into focused reusable scripts and working notes for later promotion into the playbook.Scripts should be small, readable, and parameterized rather than tied to incident-specific resource names.Source: A Responder Prompt · p. 16
- Mitigation is complete only after readiness, representative end-to-end traffic paths, and system internals are verified healthy; source-controlled causes must be fixed in source and redeployed.The prompt also directs the agent to keep diagnosing concurrent faults and prioritize mitigation while ensuring all relevant paths and internals recover.Source: A Responder Prompt · p. 16
Reflect and Update Memory
- After signaling completion, the responder updates detectors and writes a generic playbook and routing index grounded in evidence from the live cluster and source tree.If preflight findings helped, revise the existing detector or playbook rather than creating an overlapping duplicate.Source: A Responder Prompt · p. 17
- A reusable playbook should parameterize resource roles, ground each step in observed evidence, explain the cause-to-symptom chain, include a discriminating check, and localize faults across dependencies.Its README captures signals, disambiguation, triage, fault localization, verification evidence, and mitigation; scripts provide parameterized procedural detail.Source: A Responder Prompt · p. 17
- Detector updates should match confirmed causes to helpful playbooks, correct misrouting or coverage gaps, and use recurring patterns in the outcomes log to justify structural changes.A detector that repeatedly fires without helping a confirmed cause should be tightened or split; playbooks reached only through open-ended investigation may need detectors.Source: A Responder Prompt · p. 17
Authoring a detector
- Detectors should be resource-generic, narrow, and persistent: prefer cause-specific invariants, avoid unrelated-resource prerequisites, and do not flag transient or self-recovered symptoms.For ongoing crashes, gate checks on states such as CrashLoopBackOff or not-Ready pods with repeated restarts.Source: A Responder Prompt · p. 17
A Responder Prompt
- Detectors inspect a read-only Kubernetes snapshot and return structured findings with concise evidence, a primary resource, and optional playbook routing and metadata.The service-endpoints example checks for services with selectors but no ready endpoints and includes relevant pod and service details.Source: A Responder Prompt · pp. 18–19
- Findings may also describe suspicious state without routing to a playbook, using evidence and metadata to explain the detected condition.The example flags pods by phase and provides a hint to inspect events, container statuses, logs, and the owning workload.Source: A Responder Prompt · p. 19
- Detectors should generalize across resource instances while checking one specific recurring fault, and route only to playbooks directly relevant to the detected symptom.Keep playbooks self-contained, avoid overlapping diagnosis paths, and maintain explicit diagnosis-to-playbook mappings when detectors or checks change.Source: A Responder Prompt · p. 19
- Keep `.sdo/` predictable and maintainable, separating typed diagnostics, playbook guidance and helpers, and real application manifests or deployment source.The recommended layout places detectors under `diagnostics/` and issue-specific documentation and optional scripts under `playbooks/`; commit `.sdo/` changes without copying application files there.Source: A Responder Prompt · pp. 19–20