Source-linked AI summary
Can AI Remediate Backend Failures Safely? GuardedAct with Blast-Radius-Aware Sandboxing
Wanrong Cai, Tianyu Yu, Shaorui Pi, Xiaoxuan Sun, Wenrui Ma
TL;DR
LLM-generated remediation actions can damage dependent services when executed without safety verification. GuardedAct inserts digital-twin simulation, blast-radius estimation, and rollback-confidence gating before production execution, achieving 87.4% recovery while reducing collateral damage from 25.6% to 5.2% relative to direct LLM remediation. The framework trades this safety improvement for approximately 8 seconds of additional MTTR and remains evaluated only at benchmark scale.
Problem
Directly executing LLM-generated repair actions in production risks cascading collateral damage in microservice systems.
Method
GuardedAct generates typed candidate actions, simulates them in a digital-twin sandbox, estimates blast radius, and applies a rollback-confidence gate before production execution.
Results
87.4% overall recovery and collateral damage reduced from 25.6% to 5.2% relative to direct LLM remediation.
Takeaways & Limitations
Sandbox verification and rollback-confidence gating improve remediation safety while preserving a competitive recovery rate.
Takeaways & Limitations
The evaluation is limited to one benchmark with five fault types, and the digital twin cannot capture all runtime non-determinism.
Abstract
from arXiv · showhide
Large Language Models (LLMs) have shown promising capabilities in generating remediation actions for microservice failures. However, directly executing AI-generated repair actions in production risks cascading collateral damage. We propose GuardedAct, a sandbox-first remediation framework that interposes a blast-radius-aware verification layer between the LLM action generator and the production environment. GuardedAct operates in four phases: (1) ingesting a diagnosis report together with the live system topology and recent telemetry, (2) prompting an LLM to produce a ranked list of candidate remediation actions, (3) simulating each action in a lightweight digital-twin sandbox that estimates the blast radius and assigns a risk label, and (4) enforcing a rollback-confidence gate that auto-executes only low-risk actions while escalating high-risk ones for human review. We evaluate GuardedAct on five fault scenarios injected into the DeathStarBench social-network application. Experimental results show that GuardedAct achieves an overall recovery rate of 87.4% while reducing collateral damage by 79.7% relative to direct LLM execution (from 25.6% to 5.2%), at the cost of a modest sandbox-induced increase in mean time to recovery (approximately 8 s). Ablation studies confirm that each component contributes meaningfully to the safety-speed trade-off.
I. INTRODUCTION
GuardedAct addresses the safety gap in AI-driven microservice remediation by verifying LLM-generated actions before production execution. It combines sandboxing, blast-radius analysis, and rollback-aware gating to limit collateral damage.
- Cascading failures in microservice dependency chains motivate automated remediation, but direct execution of LLM-generated actions can cause collateral damage exceeding the original failure.
- GuardedAct treats AI-generated remediation as untrusted code and verifies candidate actions in a sandboxed digital twin before permitting production side effects.
- The framework combines dependency-graph and simulated-telemetry analysis with a rollback-confidence threshold to control the autonomy–safety trade-off.
- Prior work separates remediation generation, diagnosis, resilience probing, and behavioral containment, while GuardedAct connects these capabilities at the production execution boundary.
III. METHODOLOGY
GuardedAct separates LLM-generated remediation from production execution through typed candidate actions and a verification-oriented architecture. It combines diagnosis, topology, telemetry, ranked action generation, and downstream symbolic analysis.
- III. METHODOLOGY: GuardedAct ingests a diagnosis report, service-dependency graph, and recent telemetry before generating remediation candidates.Telemetry includes CPU utilization, memory usage, request latency, and error rates.
- III. METHODOLOGY: The LLM action generator emits a ranked sequence of candidate actions from a typed vocabulary.The vocabulary includes restart, drain, traffic shifting, rate limiting, and configuration rollback.
- III. METHODOLOGY: Typed actions represent operations and targets explicitly, enabling downstream components to reason about candidates symbolically.This interface lets the verifier assess the same symbolic candidate independently of where the generating model is hosted.
B. Blast-Radius Estimation
GuardedAct estimates remediation risk by simulating candidate actions and measuring degradation across reachable services. The blast-radius score combines dependency reachability, traffic weights, and simulated degradation probabilities.
- B. Blast-Radius Estimation: Blast radius is computed over services reachable from the target within d hops, combining traffic weights with simulated degradation probabilities.The reachability set is defined from the service-dependency graph, while traffic weights are normalized to [0,1].
- B. Blast-Radius Estimation: A service is counted as degraded when simulated 99th-percentile latency rises by more than 50% or error rate rises by more than 5 percentage points.The degradation probability is the fraction of the simulated horizon during which either condition holds.
- B. Blast-Radius Estimation: Risk labels are assigned by binning each candidate’s blast-radius score at the 33rd and 66th percentiles of a 50-incident calibration set.The resulting labels are low, medium, and high.
- B. Blast-Radius Estimation: The estimator uses replay-based sandbox simulation rather than a learned graph forecaster to anticipate dependency-mediated degradation.The hop depth is set to three because effects beyond three hops contributed less than 2% of observed degradation in a pilot study.
- B. Blast-Radius Estimation: The architecture routes candidate actions through input collection, ranked generation, sandbox verification, and rollback-confidence gating.The sandbox verification phase simulates each action in a digital twin and assigns a risk label.
C. Rollback-Confidence Gate
GuardedAct combines simulated blast radius with intrinsic action reversibility to estimate rollback confidence. An operator-configurable threshold determines whether an action is automatically executed or sent for human review.
- C. Rollback-Confidence Gate: Rollback confidence aggregates blast radius and intrinsic reversibility to reflect how safely a candidate action can be reversed without residual side effects.The score normalizes blast radius by the maximum observed among candidates in the current incident window.
- C. Rollback-Confidence Gate: Configuration rollback receives the highest reversibility score at 0.90, while restart receives the lowest at 0.10.The calibrated scores are based on 30 manually labeled recovery traces.
- C. Rollback-Confidence Gate: Actions meeting the rollback-confidence threshold are auto-executed; all others receive a human-review decision.The default operator-configurable threshold is θ=0.6.
- C. Rollback-Confidence Gate: The gate treats automated actuation as a constrained operational decision in which reversibility and simulated blast radius determine permissible exposure.Escalation consumes operator attention but is intended to prevent indiscriminate execution of generated actions.
D. Digital-Twin Sandbox
GuardedAct evaluates candidate remediation actions in a lightweight replica that propagates effects through the dependency graph before production execution. The sandbox records downstream service outcomes to support blast-radius estimation and gating decisions.
- The sandbox replays recent telemetry in containerized stubs and applies each candidate action to the targeted service.The stubs mirror the production dependency graph while reproducing the latest observed telemetry window.
- Effects propagate across downstream dependencies for a configurable horizon, producing latency, error-rate, and throughput deltas for blast-radius estimation.GuardedAct measures consequences over the replay horizon rather than only at the initially targeted service.
- Verification records system-level outcomes across dependent components because local improvement alone cannot establish that an intervention is acceptable.Downstream load and delay are considered before execution is gated.
A. Testbed and Fault Scenarios
The evaluation uses a 30-pod Kubernetes deployment of DeathStarBench’s social-network application with injected microservice fault scenarios. It compares direct, rule-based, typed, and sandbox-verified remediation methods under a shared LLM prompt and temperature.
- The DeathStarBench social-network application runs on a Kubernetes cluster with 30 microservice pods for evaluation.
- Five fault scenarios are injected, including CPU saturation, downstream timeout, memory leak, misconfiguration, and network jitter.The supplied scenario description explicitly names the first four and begins the fifth as network jitter.
- The comparison includes Direct LLM, Rule-Based, Typed without sandboxing, and the full GuardedAct pipeline.All LLM-based methods use the same prompt template and temperature T = 0.2.
C. Metrics
The study measures recovery, collateral damage, recovery time, rollback success, and human escalation using explicit health criteria. GuardedAct achieves the strongest reported recovery and collateral-damage results, with added recovery-time overhead but higher rollback success.
- 87.4% overall recovery and 5.2% collateral damage make GuardedAct the strongest reported method across the fault scenarios.The overall comparison is averaged over 100 trials per scenario.
- 64% recovery with 25.6% collateral damage characterizes Direct LLM remediation, while Rule-Based reaches 66% recovery with 19.8% collateral damage.
- Typed remediation reduces collateral damage to 14.2% by constraining the action vocabulary, but remains less safe without sandbox verification.
- 61.4 s MTTR for GuardedAct versus 52.6 s for Typed reflects 8.2 s average sandbox overhead per action.
- 90% rollback success for GuardedAct versus 55% for Direct LLM is attributed to pre-validating rollback feasibility before committing actions.
B. Per-Scenario Analysis
GuardedAct combines strong recovery with low collateral damage, while its safety depends on all three pipeline components and a calibrated rollback-confidence threshold. The approach trades modest latency and benchmark-limited robustness for safer automated remediation.
- Per-scenario comparison: GuardedAct occupies the high-recovery, low-MTTR region, while Direct LLM reaches only 55%–72% recovery and causes 35% damage in the memory-leak scenario.The memory-leak damage cascades through three downstream services after an aggressive restart.
- Ablation study: Removing the sandbox lowers recovery by 10–12 points in memory-leak and misconfiguration cases, while removing the rollback-confidence gate causes 14–20-point declines.The ablation results indicate that unchecked high-risk actions can trigger secondary failures.
- Threshold sensitivity: Increasing the threshold from 0.3 to 0.9 reduces harm from 12% to below 2% but lowers recovery from 94% to 65% and raises intervention from 5% to 80%.The recommended range [0.55,0.70] keeps harm below 4% while balancing recovery and intervention.
- Overhead and limitations: The sandbox adds 8.2 seconds on average, and sub-second recovery scenarios may require a lighter simulation model.The overhead is presented as a limitation for latency-critical settings.
- Overhead and limitations: Noisy upstream diagnoses remain a boundary: sandboxing can catch large simulated blast radii, but a safe-looking action targeting the wrong service may still execute.The paper identifies robustness to noisy RCA outputs as future work.
VII. CONCLUSION
The conclusion presents GuardedAct as a blast-radius-aware sandboxing framework that improves the safety of AI-generated microservice remediation. It reports lower collateral damage with competitive recovery, while noting modest latency overhead and benchmark-scale evaluation limits.
- VII. CONCLUSION: GuardedAct reduces collateral damage from 25.6% to 5.2% while maintaining an 87.4% recovery rate.The comparison is reported for the framework’s benchmark evaluation.
- VII. CONCLUSION: The sandbox, blast-radius predictor, and rollback-confidence gate each contribute to the reported safety improvement.Ablation studies support the contribution of all three components.
- VII. CONCLUSION: GuardedAct adds approximately 8 seconds of MTTR overhead, and the findings remain limited to benchmark-scale environments requiring larger industrial validation.The conclusion frames the overhead as acceptable in safety-sensitive settings.