Source-linked AI summary
Chaos Engineering
Ali Basiri, Niosha Behnam, Ruud de Rooij, Lorin Hochstein, Luke Kosewski, Justin Reynolds, Casey Rosenthal
TL;DR
Chaos Engineering addresses the difficulty of verifying reliability in complex distributed systems whose failures traditional testing may miss. The paper defines a practice of experimenting with real-world or simulated events, often in production, to build confidence in system behavior. Its central conclusion is that automated, repeated experiments can maintain confidence, uncover weaknesses, and support improvements, while selecting event combinations remains an open challenge.
Problem
Traditional testing cannot sufficiently identify distributed-system failures involving service interactions, and complete production architectures cannot always be reproduced in test contexts.
Method
The paper defines Chaos Engineering as experiments that vary real-world events, measure steady-state behavior, and automate experiments to run continuously.
Results
Automated Chaos Engineering experiments can maintain confidence that production services withstand failures, while experiments can also uncover weaknesses suggesting improvement.
Takeaways & Limitations
Chaos Engineering is presented as an emerging discipline for applying empirical experimentation to improve confidence in software-system availability.
Takeaways & Limitations
The space of possible event combinations is potentially large, and the paper leaves open how to decide which experiments to run.
Abstract
from arXiv · showhide
Modern software-based services are implemented as distributed systems with complex behavior and failure modes. Many large tech organizations are using experimentation to verify the reliability of such systems. We use the term "Chaos Engineering" to refer to this approach, and discuss the underlying principles and how to use it to run experiments.
Build a hypothesis around steady state behavior
Chaos Engineering hypotheses target steady-state, user-visible availability metrics rather than fine-grained internal measures. At Netflix, SPS captures streaming starts and normally varies predictably, making unexpected changes actionable.
- Build a hypothesis around steady state behavior: Netflix focuses Chaos Engineering experiments on availability as experienced at the system boundary.Internal service failures do not necessarily affect overall availability, so experiments emphasize user-facing steady-state behavior.
- Build a hypothesis around steady state behavior: SPS, or stream starts per second, measures how many users begin streaming video each second.It operationalizes whether users can find content and successfully watch it.
- Build a hypothesis around steady state behavior: SPS varies slowly and predictably over a day, so an unexpected change signals a system problem.Netflix engineers use familiarity with normal SPS variation to distinguish expected fluctuations from concerning changes.
- Build a hypothesis around steady state behavior: Steady-state availability metrics differ by domain, such as completed purchases per second for ecommerce services.The metric should have a direct link to the service’s availability for users.
- Build a hypothesis around steady state behavior: Experiments form hypotheses about how a treatment will affect the chosen steady-state metric.For regional failover, Netflix hypothesizes that redirecting traffic will have minimal impact on SPS.
Vary real‐world events
Chaos experiments should sample disruptive inputs from real-world conditions rather than test only the software’s happy path. Events may be injected or simulated, with realism balanced against potential system harm.
- Vary real‐world events: Testing only the happy path misses error conditions and corner cases that occur in real systems.Observed examples include malformed requests, exhausted resources, latency spikes, and unexpected traffic surges.
- Vary real‐world events: Chaos experiments choose stimuli from the space of possible real-world inputs, including historical outage events.Any input that could disrupt steady-state behavior can be considered a candidate.
- Vary real‐world events: Netflix experiments include terminating virtual machines, injecting latency, failing requests or services, and making an Amazon region unavailable.These examples cover infrastructure, network, service, and regional failure conditions.
- Vary real‐world events: Experiments can also vary request rates, runtime parameters, and metadata propagated through the system.Netflix notes that useful inputs need not be limited to hardware and software failures.
- Vary real‐world events: When direct injection is infeasible, engineers simulate the event while balancing realism against risk.Netflix simulates regional failure by redirecting requests to other regions rather than taking an entire region offline.
Run experiments in production
Distributed-service failures often arise from interactions that traditional testing cannot fully identify or reproduce. Chaos Engineering therefore uses integration-focused experiments in production, where the deployed system and real clients expose behaviors unavailable in complete test replicas.
- Run experiments in production: Netflix cannot fully reproduce its entire architecture for an end-to-end test.The system’s scale and distributed deployment make complete reproduction impractical.
- Run experiments in production: Traditional software testing is insufficient for identifying potential distributed-system failures.The paper motivates production experimentation with complex failure modes involving interactions among services.
- Run experiments in production: Some distributed failure modes require integration testing because they involve interactions among services.Examples include unbounded queues causing client failure and incorrectly cached transient errors spreading to other clients.
- Run experiments in production: Production experiments remain necessary even when the entire system can be reproduced in a test context.Synthetic clients and DNS configuration can still differ from production conditions.
Automate experiments to run continuously
Chaos Engineering experiments must be automated and repeated because continuously changing systems make past results less reliable. Repeated experiments maintain confidence by testing whether steady-state behavior persists under injected failures.
- Automate experiments to run continuously: Automating experiments maintains confidence in results as the system evolves and past experiment findings become less reliable.Netflix reports that services, configurations, and metadata change continuously, reducing confidence in earlier experiments over time.
- Automate experiments to run continuously: Experiments can begin manually but should ultimately run repeatedly, with cadence determined by context.Netflix runs Chaos Monkey continuously on weekdays and Chaos Kong once a month.
- Automate experiments to run continuously: Automation is feasible and helps ensure that new production services continue to withstand the failures targeted by the experiments.Netflix’s experience with Chaos Monkey supports continuously running automated failure experiments.
- Automate experiments to run continuously: An experiment defines steady state, hypothesizes that it persists in control and experimental groups, injects real-world variables, and tests for differences.The procedure compares measurable system outputs after introducing failures such as crashed servers or severed network connections.
- Automate experiments to run continuously: At experiment completion, unchanged behavior increases confidence in system resilience, while a weakness suggests a path for improvement.The outcome is evaluated at the system boundary by examining whether the chosen steady-state behavior changes.