PaperlayerView exact revision on arXiv

Source-linked AI summary

LLM-Powered Fully Automated Chaos Engineering: Towards Enabling Anyone to Build Resilient Software Systems at Low Cost

Daisuke Kikuta, Hiroki Ikeuchi, Kengo Tajiri

arXiv:2511.07865v1cs.SEcs.AIcs.CLcs.MA

TL;DR

Complex dependencies in distributed systems make system-wide behavior difficult to predict, while several CE processes remain candidates for automation. ChaosEater uses a predefined LLM-agent workflow to automate CE cycles for Kubernetes systems. Case studies report low-cost cycles that evaluators judged reasonable, alongside stated scope limitations.

  • Problem

    Complex dependencies among distributed services can make faults propagate into unpredictable system behavior, and full CE automation is only becoming feasible.

    Source: §I · p. 1

  • Method

    ChaosEater assigns subdivided CE processes to LLMs in a predefined workflow, targeting Kubernetes systems and using software-engineering tasks to complete cycles.

    Source: §I · pp. 1–2

  • Results

    NGINX costs were $0.21 and 11min, while SOCKSHOP costs were $0.84 and 25min; all evaluators rated every phase above the positive-rating threshold for both systems.

    Source: Costs and stability · p. 4, Qualitative validation · p. 5

  • Takeaways & Limitations

    The case studies provide evidence for the feasibility of fully automating CE, suggesting that anyone may build resilient systems at low cost.

    Source: §IV · p. 5

    Current ChaosEater can be deployed only in development environments due to security concerns, supports only Kubernetes-manifest reconfiguration, and has vulnerability discovery limited to a single CE cycle.

    Source: §IV · p. 5

Abstract

from arXiv · show

Chaos Engineering (CE) is an engineering technique aimed at improving the resilience of distributed systems. It involves intentionally injecting faults into a system to test its resilience, uncover weaknesses, and address them before they cause failures in production. Recent CE tools automate the execution of predefined CE experiments. However, planning such experiments and improving the system based on the experimental results still remain manual. These processes are labor-intensive and require multi-domain expertise. To address these challenges and enable anyone to build resilient systems at low cost, this paper proposes ChaosEater, a system that automates the entire CE cycle with Large Language Models (LLMs). It predefines an agentic workflow according to a systematic CE cycle and assigns subdivided processes within the workflow to LLMs. ChaosEater targets CE for software systems built on Kubernetes. Therefore, the LLMs in ChaosEater complete CE cycles through software engineering tasks, including requirement definition, code generation, testing, and debugging. We evaluate ChaosEater through case studies on small- and large-scale Kubernetes systems. The results demonstrate that it consistently completes reasonable CE cycles with significantly low time and monetary costs. Its cycles are also qualitatively validated by human engineers and LLMs.

I. INTRODUCTION

ChaosEater applies LLMs to automate the full chaos-engineering cycle, addressing manual work that remains labor-intensive and expertise-intensive. Case studies report reasonable cycles at low time and monetary cost, with qualitative validation.

Motivation

  • Existing tools automate experiment execution and analysis, but hypothesis formation, experiment planning, and system reconfiguration remain manual and labor-intensive.The paper says automating these processes requires multi-domain expertise.Source: §I · p. 1
  • LLMs are proposed for these tasks because CE for software systems corresponds to software engineering, where LLM capabilities are considered suitable for the manual processes.The paper also notes concurrent exploration of LLMs for partial software-management processes.Source: §I · p. 1

Contributions

  • The paper’s workflow uses LLM agents and existing tools to complete the systematic CE cycle.Figure 1 illustrates representative agent inputs and outputs; both depicted Kubernetes clusters refer to the same cluster.Source: Figure 1 · p. 2

Evaluation

  • ChaosEater completes reasonable single CE cycles on small- and large-scale Kubernetes systems for $0.2–0.8 and 11–25min, with validity confirmed by human engineers and LLMs.The paper evaluates cost and stability quantitatively and CE-process validity qualitatively.Source: §I · p. 2

Contributions

  • ChaosEater is presented as the first LLM-based system to automate the entire CE cycle through a predefined workflow of subdivided processes.The paper releases its resources as a foundation for subsequent work.Source: §I · pp. 1–2
  • The paper presents full-cycle automation as evidence that system-resilience improvement may be feasible to automate.It also releases ChaosEater’s resources for subsequent work.Source: §I · p. 2

II. PROPOSED SYSTEM: CHAOSEATER

ChaosEater takes user instructions and Kubernetes deployment files, runs a six-phase CE workflow, and returns a cycle summary and reconfigured files.

Workflow overview

  • The system takes optional CE-cycle instructions, Kubernetes manifests, and a Skaffold configuration file as input.It deploys the user’s system to a Kubernetes cluster using the Skaffold configuration.Source: §II · p. 2
  • Its six phases are pre-processing, hypothesis, experiment, analysis, improvement, and post-processing.Source: §II · p. 2
  • ChaosEater returns a summary of the completed cycle and a modified folder of reconfigured manifests with the corresponding Skaffold configuration.Each LLM agent has a specific role in a subdivided CE process, and the underlying LLMs require no additional fine-tuning.Source: §II · p. 2

A. Phase 0: Pre-processing

Pre-processing deploys the user’s Kubernetes system and fills in implicit context for later CE phases.

A. Phase 0: Pre-processing

  • ChaosEater first deploys the user’s system to the Kubernetes cluster by running the Skaffold configuration file.Source: A. Phase 0: Pre-processing · p. 2
  • LLM agents sequentially fill in context, including manifest summaries, potential resilience and redundancy issues, and a possible application.The pre-processing phase also filters harmful prompts in user instructions.Source: A. Phase 0: Pre-processing · pp. 2–3
  • Kubernetes manifests define system resources such as microservices, while the Skaffold configuration defines how those resources are deployed to a cluster.Source: A. Phase 0: Pre-processing · p. 2

B. Phase 1: Hypothesis

The hypothesis phase specifies resilience requirements for an assumed failure scenario by defining steady states and validating them with unit-test scripts.

B. Phase 1: Hypothesis

  • ChaosEater defines the required system resilience for an assumed failure scenario by first defining steady states and then the failure scenario.Source: B. Phase 1: Hypothesis · p. 3

Steady-state definition:

  • Each steady state pairs a measurable system value with a threshold and is satisfied when the value meets that threshold.An agent selects an application-critical state, inspects its current value through the Kubernetes API or k6, and sets a threshold that holds under normal conditions.Source: Steady-state definition · p. 3
  • An agent adds threshold-based assertions to an inspection script, producing a unit-test script that mechanically validates steady states; the paper calls this Validation as Code (VaC).Agents repeat the process to define multiple nonduplicative steady states until their number is judged sufficient.Source: Steady-state definition · p. 3

Failure definition:

ChaosEater defines failure scenarios as sequences of faults that simulate possible system events. Its experiment phase plans and executes tests of the hypothesis.

Failure definition:

  • An LLM agent proposes a possible failure scenario from preprocessed user inputs and steady states, then defines faults that simulate it.Chaos Mesh support constrains the available faults; another agent refines each fault’s scope, subtype, and strength into structured parameters.Source: Failure definition · p. 3
  • Fault parameters are refined stepwise, helping LLMs specify accurate structured JSON outputs.The parameters include injection scope, fault subtype, and fault strength.Source: Failure definition · p. 3

C. Phase 2: (Chaos) Experiment

Experiment planning:

ChaosEater plans scheduled validation and fault injection, then updates the plan after system reconfiguration while preserving the experiment’s intent.

Experiment planning:

  • Agents schedule pre-validation, fault injection, and post-validation stages, selecting VaC scripts, faults, timings, and durations from user inputs and the hypothesis.The plan becomes a Chaos Mesh workflow manifest for scheduled fault injection and hypothesis validation.Source: Experiment planning · p. 3

Experiment replanning:

  • After manifest changes, agents update VaC inspection targets and fault scopes, making minor adjustments without changing the experiment’s original intent.A new Chaos Mesh workflow manifest is generated by updating corresponding parts of the previous one.Source: Experiment replanning · pp. 3–4

Experiment execution:

  • ChaosEater applies the workflow manifest to the Kubernetes cluster, where Chaos Mesh automatically runs scheduled fault injections and hypothesis validation.ChaosEater waits for the experiment to complete.Source: Experiment execution · p. 4

D. Phase 3: Analysis

Failed validation triggers LLM analysis and manifest reconfiguration; ChaosEater repeats the experiment, analysis, and improvement phases until the hypothesis is satisfied.

D. Phase 3: Analysis

  • When VaC scripts fail, an LLM agent analyzes manifests, experiment timing, and failure logs to identify causes and report countermeasures.If all scripts pass, ChaosEater finishes the cycle and moves to post-processing.Source: D. Phase 3: Analysis · p. 4

E. Phase 4: Improvement

  • An LLM agent reconfigures Kubernetes manifests by replacing, creating, or deleting them until all VaC scripts pass, gradually increasing redundancy to avoid excess.The agent uses the hypothesis, experiment plan, and improvement-loop history.Source: E. Phase 4: Improvement · p. 4

Improvement loop:

  • ChaosEater repeats experiment, analysis, and improvement until the hypothesis is satisfied; loop history suppresses repeated reconfigurations.The history contains experimental results, analysis reports, and reconfigurations.Source: Improvement loop · p. 4

F. Extra Phase: Post-processing

After a cycle, ChaosEater summarizes its phases and provides the reconfigured manifests and configuration file. The case study evaluates cycle quality and costs on two Kubernetes systems.

F. Extra Phase: Post-processing

  • An LLM agent summarizes user input and the four completed phases, and ChaosEater delivers that summary with reconfigured manifests and a Skaffold configuration file.Source: F. Extra Phase: Post-processing · p. 4

III. CASE STUDY

  • The case study evaluates whether CE processes are meaningful, alongside whether reconfigurations are appropriate, using quantitative and qualitative case studies.The evaluation focuses on two critical cases.Source: §III · p. 4
  • Table I reports time and monetary costs for single ChaosEater cycles across the target systems.Source: Table I · p. 4
  • NGINX is a two-manifest minimal system with restartPolicy set to Never; SOCKSHOP has 29 manifests, with two replicas except for its single-replica front end.These configurations introduce resilience issues for the case study.Source: §III · p. 4
  • The cases test whether ChaosEater identifies and addresses resilience issues; each system is run through five single-cycle trials using gpt-4o-2024-08-06 at temperature zero.Inputs are limited to a one-minute experiment instruction, with access methods also supplied for SOCKSHOP.Source: §III · p. 4

Costs and stability:

ChaosEater completed cycles on both target systems with low reported time and monetary costs, and it completed all five runs without runtime errors. Reconfiguration succeeded in all NGINX runs and four of five SOCKSHOP runs.

Costs and stability:

  • $0.21 and 11min for NGINX versus $0.84 and 25min for SOCKSHOP; despite more than ten times the resources, SOCKSHOP’s cost rose about fourfold and its time doubled.The authors note that they lack comparable statistical data on human engineers’ working time and labor costs.Source: Costs and stability · p. 4
  • Across five runs per system, ChaosEater completed every cycle without runtime errors and reconfigured NGINX five times and SOCKSHOP four times.Source: Costs and stability · p. 4
  • Figure 3 treats scores of 3 or higher as positive ratings for CE cycles on each system.Source: Figure 3 · p. 5
  • A valid CE cycle was completed even in the case that required no reconfiguration.Source: Costs and stability · p. 5

Qualitative validation:

In one selected run per target system, evaluators rated every CE phase above the positive threshold. The paper concludes that full CE automation is feasible, while identifying limits in deployment setting, reconfiguration scope, and exploration across cycles.

Qualitative validation:

  • All evaluators rated every phase above the positive threshold for both systems in the selected cycles.The five-point scale counted scores of 3 or higher as positive; evaluators were two external engineers and three LLMs.Source: Qualitative validation · p. 5

IV. CONCLUSION, LIMITATIONS, AND FUTURE WORK

  • The study presents evidence for the feasibility of full CE automation as a route toward building resilient systems at low cost.The authors motivate automation by CE’s labor demands and need for multi-domain expertise.Source: §IV · p. 5
  • Current ChaosEater is limited to development environments, Kubernetes-manifest reconfiguration, and vulnerability discovery within a single CE cycle.Source: §IV · p. 5
  • Future work would extend reconfiguration to frontend, application, and lower-layer code, and explore vulnerabilities across multiple cycles with iteratively refined hypotheses.Source: §IV · p. 5
Loading 2511.07865v1…