PaperlayerView exact revision on arXiv

Source-linked AI summary

A Hybrid Rule-Based and AI-Augmented Framework for Automatic Failure Recovery in DevOps Deployments

Raju Chowdhury, Aniruddha Singh Gautam, Apurv Ghai

arXiv:2609.26838v1cs.SEcs.DC

TL;DR

DevOps failure recovery is challenged by limitations in rule-based and AI-based approaches, including limited transparency and autonomy. The paper combines deterministic recovery rules with machine-learning fault prediction and evaluates the framework on a synthetic distributed-system log dataset. It reports recovery and operational improvements over a rule-based baseline, while noting the evaluation's limited scope.

  • Problem

    Existing DevOps recovery approaches are described as often relying on AI or LLM-based methods with human intervention and limited transparency, while constrained evaluations and prediction-focused methods leave a gap in autonomous recovery.

    Source: §I · p. 1, §II · p. 2

  • Method

    The framework integrates machine-learning fault prediction into a deterministic rule-based recovery pipeline for DevOps deployments.

    Source: §I · p. 1, §V · p. 6

  • Results

    The framework achieved 83.3% total recovery success; against a deterministic rule-based baseline, it reported over 94.6% reduction in failures, downtime and MTTR and 7,462 dollars saved per year.

    Source: §IV · p. 5

  • Takeaways & Limitations

    The experimental results support the feasibility of hybrid intelligent automation in DevOps systems, with increased prediction accuracy and recovery rates and lower MTTR and downtime than conventional approaches.

    Source: §V · p. 6

    Evaluation used a synthetic dataset and four fault categories, limiting generalizability to other failure scenarios.

    Source: A. Discussion · p. 6

Abstract

from arXiv · show

Automatic failure recovery is another difficult area of DevOps deployments due to the complexity of the system, high workload and the limitations of the traditional rules-based or manual approach. This study proposes a hybrid model integrating deterministic, rule-based recovery with the assistance of machine learning by fault prediction to automatically monitor failures, identify, classify and recover failures in real time. To test the architecture, a distributed log dataset of 100,000 records was used for models such as Decision Tree, Random Forest, Logistic Regression, LightGBM, Autoencoder with BiLSTM. The Decision Tree had a strong performance in defect identification with an F1 score of 80.8%, recall of 80.0%, precision of 81.6%, and accuracy of 89.4%. An impressive 83.3% success rate was achieved by the automated recovery measures, resulting in a 94.6% decrease in mean time to recovery (MTTR), a 95.4% reduction in downtime, and annual savings of 7,462. The combination of explainable rule-based logic and adaptive AI prediction in the framework means it is more resilient, operationally efficient, and scalable, and an effective, autonomous approach to DevOps environments evaluated under controlled experimental conditions.

I. INTRODUCTION

The paper frames DevOps failure recovery as a gap between deterministic rules and AI-based prediction, and proposes combining them to address predictable and dynamic failures. It also situates the work among AI-augmented recovery studies and identifies limited transparency, autonomy, and real-world evaluation as shortcomings in prior approaches.

I. INTRODUCTION

  • The proposed hybrid combines deterministic recovery for predictable failures with machine-learning fault prediction for unknown or dynamic failures.AI augmentation is defined as machine-learning-assisted fault prediction within a deterministic rule-based recovery pipeline.Source: §I · p. 1
  • The paper presents contributions spanning hybrid recovery, predictive and operational evaluation, scalability, explainability, and cost-effectiveness.The listed indicators include MTTR, downtime, recovery success, system stability, and operational cost savings.Source: §I · p. 1

A. Structure of Paper

  • The paper is organized into related work, the proposed framework and approach, experimental setup and results, and a conclusion with future directions.Source: A. Structure of Paper · p. 1

II. LITERATURE REVIEW

  • Related work includes a SAPAL-loop CI/CD framework, an LLM and deep-reinforcement-learning recovery mechanism reporting a 37% recovery-time reduction, and AI-enhanced DevOps and self-healing systems,.The latter reports include deployment-frequency, MTTR, change-failure-rate, detection, and automated-recovery measures.Source: §II · pp. 1–2
  • Other cited studies report a 41.7% MTTR improvement and 40% release-velocity increase, while a Kubernetes-based recovery system averaged 27 seconds across ten failure modes.Source: §II · p. 2
  • Prior approaches are described as often relying on AI or LLM-based recovery, requiring human intervention, lacking transparency or governance, and emphasizing constrained setups or prediction over end-to-end recovery.The identified gap motivates combining rule-based and AI-based automation for autonomous recovery.Source: §II · p. 2

III. METHODOLOGY

The methodology presents a hybrid system in two stages, from model development and deployment to real-time monitoring and recovery. Its workflow connects log processing and model selection with a live prediction-and-recovery pipeline.

III. METHODOLOGY

  • The framework has two stages: Model Development & Deployment, followed by Real-Time Monitoring & Recovery.Fig. 1 depicts the workflow of the hybrid AI and rule-based failure recovery framework.Source: §III · p. 2, Figure 1 · p. 2
  • The development stage preprocesses logs, splits data into 70% training and 30% testing, balances and scales it, trains multiple models, evaluates them, and deploys the selected model.Models include LR, DT, RF, LightGBM, and Autoencoder with BiLSTM; evaluation uses accuracy, precision, recall, and F1-score.Source: §III · p. 2
  • In the live stage, logs pass through AI fault prediction, rule-based classification, failure mapping and recovery-action selection, and automated execution.Performance monitoring, impact analysis, and reporting support continuing operational assessment.Source: §III · p. 2
  • Fig. 2 is a correlation heatmap of dataset attributes and engineered features.Source: Figure 2 · p. 3

A. Data Gathering

The study uses a 100,000-record distributed-system log dataset and describes preprocessing steps intended to make raw logs suitable for predictive modelling. These include handling incomplete records and standardizing timestamps.

A. Data Gathering

  • The dataset contains 100,000 log records and nine attributes, with additional engineered features derived for fault prediction.The original fields include timestamps, log levels, service names, message types, request identifiers, user identifiers, client IP addresses, and request execution times.Source: A. Data Gathering · p. 3

B. Data Preprocessing

  • Preprocessing is described as preparing raw log data for reliability, consistency, and predictive modelling.Source: B. Data Preprocessing · p. 3

1) Missing-Value Handling

2) Timestamp Normalization

  • Timestamp normalization converts datetime fields into a consistent format for temporal sequencing and analysis of failure and recovery events.Source: 2) Timestamp Normalization · p. 3

3) Duplicate Removal

The preprocessing and feature-preparation stages remove duplicate data, derive additional predictors, structure log fields, and encode categorical attributes for model use.

3) Duplicate Removal

  • Duplicate removal eliminates repeated or inconsistent records associated with remonitoring or redeploying the same code.The stated purpose is to reduce noise in the dataset.Source: 3) Duplicate Removal · p. 3

4) Feature Engineering

  • Feature engineering adds variables such as severity, failure status, failure frequency, downtime, recovery time, success indicators, and system-state flags.Source: 4) Feature Engineering · p. 3

5) Feature Extraction

  • Feature extraction converts raw log fields—including events, error codes, component IDs, timestamps, recovery actions, and outcomes—into structured modelling variables.Source: 5) Feature Extraction · p. 3

6) Label Encoding

  • Label encoding assigns categorical values, such as log level and service name, distinct integers so models can process them.Source: 6) Label Encoding · p. 3

C. Data Splitting

The data is split into training and testing sets, then training data is balanced with SMOTE and features are standardized. These steps prepare the data for model training.

C. Data Splitting

  • 70% of the data is used for training and 30% for testing.Models are trained on the training set and tested on the testing set.Source: C. Data Splitting · p. 3

D. Data Balancing Using SMOTE

E. Scaling Using StandardScaler

F. Machine Learning Models

The study trains and assesses five model types for defect prediction, describing the Decision Tree in detail. Its tree splits feature space recursively, using entropy and information gain in its formulation.

F. Machine Learning Models

  • Logistic Regression, Decision Tree, Random Forest, LightGBM, and an Autoencoder with Bi-LSTM are trained and assessed for defect prediction.Source: F. Machine Learning Models · p. 3
  • A Decision Tree recursively partitions feature space into homogeneous regions using internal decision nodes and terminal leaves.Each internal node applies a decision function to determine the traversal path for an input sample.Source: F. Machine Learning Models · p. 3
  • Information gain for a feature at a node is defined as node entropy minus the sum of its children’s entropy after splitting.The entropy formulation uses two classes: normal and failure.Source: F. Machine Learning Models · p. 3

1) Model Training

The Decision Tree is selected for its classification performance and interpretability, then deployed in a monitoring and recovery process. Training uses constrained tree growth and pruning.

1) Model Training

  • The Decision Tree uses Gini Index splitting, maximum depth 10, minimum split samples 2 and leaf samples 5, and cost-complexity post-pruning with regularization parameter alpha=0.01.The depth limit is set to avoid overfitting, while recursive partitioning distinguishes normal from failure states.Source: 1) Model Training · p. 3

G. Best Model Selection

  • The Decision Tree is selected for the highest classification performance and interpretability, making it more suitable for automated DevOps than more complex ensemble models.Its transparent decision rules map predicted faults directly to recovery actions.Source: G. Best Model Selection · pp. 3–4

H. Failure Detection and Recovery Process

  • The system monitors production logs every minute, predicts potential failures, classifies them with predefined rules, and automatically executes mapped recovery actions.Recovery rules are applied after fault prediction, not during training; monitoring and periodic retraining support adaptive responses.Source: H. Failure Detection and Recovery… · p. 4

I. Evaluation Metrics

The evaluation covers classification, recovery, operational reliability, and economic impact. The stated measures include classification scores, recovery timing and success, stability, downtime, failure count, savings, and ROI payback.

I. Evaluation Metrics

  • The evaluation uses recovery success rate and recovery time, plus accuracy, precision, recall, and F1-score for classification.Source: §I · p. 4
  • System reliability is assessed with stability, average downtime, MTTR, and failure count; economic impact uses cost savings and ROI payback period.Source: §I · p. 4
  • TP, TN, FP, and FN denote classification outcomes, while Nf counts failure events and tf,i and tr,i mark each event’s occurrence and recovery completion times.Source: §I · p. 4

IV. EXPERIMENTAL RESULTS AND ANALYSIS

Under controlled tests on synthetic distributed-system logs, the hybrid framework improved recovery and operational measures over a no-ML rule-based baseline, while the Decision Tree achieved 89.4% accuracy.

IV. EXPERIMENTAL RESULTS AND ANALYSIS

  • The evaluation is a proof-of-concept under controlled conditions using a synthetic distributed-system log dataset.The setup used Python 3.11 on Ubuntu 22.04 LTS.Source: §IV · p. 4
  • 89.4% accuracy was achieved by the Decision Tree, with balanced precision, recall, and F1-score across the compared models.Table II compares models using the same train-test partition and preprocessing pipeline.Source: Table II · p. 4, §IV · p. 5
  • 83.3% total recovery success was reported, including 90% success for automated redeployment and 75% for cache clearing.The rates came from a fault-injection simulation comparing predicted fault-driven redeployments with expected redeployments in the fault-response mapping table.Source: Table III · p. 5, §IV · p. 5
  • Retraining is offline at scheduled intervals rather than online or continual, and reported performance gains reflect incremental incorporation.Figure 4 presents performance across retraining cycles.Source: Figure 4 · p. 5, §IV · p. 5
  • Against the no-ML rule-based baseline, the hybrid system reports over 94.6% reductions in failures, downtime, and MTTR, 28.2% higher recovery success, 4.3% higher stability, and $7,462 saved yearly.The comparison used the same synthetic dataset, preprocessing, recovery rules, and evaluation protocol.Source: Table IV · p. 5, §IV · p. 5
  • The hybrid method shows smaller, more predictable MTTR, minimizes 24-hour cumulative downtime, and maintains availability above SLA standards.The MTTR distribution is compared with the conventional DevOps model; downtime and availability are compared with conventional systems and SLA standards, respectively.Source: Figure 5 · p. 5, §IV · pp. 5–6, Figure 6 · p. 6

A. Discussion

The Decision Tree performed well on the synthetic benchmark, but the results demonstrate feasibility of hybrid recovery rather than superiority in operational DevOps conditions. Limited fault diversity and statistical validation constrain the findings.

A. Discussion

  • The Decision Tree showed good fault-classification performance, while Random Forest and LightGBM had no clear advantage on this dataset.The benchmark’s predefined fault characteristics simplify classification compared with real production environments.Source: A. Discussion · p. 6
  • The synthetic evaluation covered four fault categories, limiting generalizability to scenarios such as network partitions, cascading failures, configuration drift, and security failures.The study also notes heterogeneous workload failures among untested scenarios.Source: A. Discussion · p. 6
  • The evaluation lacked repeated trials, k-fold cross-validation, confidence intervals, and significance testing, so results are proof-of-concept rather than production-deployment evidence.Source: A. Discussion · p. 6

V. CONCLUSION AND FUTURE STUDY

The paper concludes that combining machine-learning fault prediction with deterministic recovery supports efficient failure detection, classification, and remediation with minimal human intervention. It presents hybrid automation as feasible while identifying fault diversity and statistical power as areas for future work.

V. CONCLUSION AND FUTURE STUDY

  • Machine-learning fault prediction combined with deterministic recovery supports fault detection, classification, and remediation with minimal human intervention.Source: §V · p. 6
  • Experimental results support the feasibility of hybrid intelligent automation, with reported higher recovery rates and lower MTTR and downtime than conventional DevOps approaches.Source: §V · p. 6
  • Future work will add more complex fault categories and statistical testing, and may explore reinforcement learning, multi-agent coordination, and semantic-log interpretation using an LLM.Source: §V · p. 6
Loading 2609.26838v1…