Source-linked AI summary
LADs: Leveraging LLMs for AI-Driven DevOps
Ahmad Faraz Khan, Azal Ahmad Khan, Anas Mohamed, Haider Ali, Suchithra Moolinti, Sabaat Haroon, Usman Tahir, Mattia Fazzini, Ali R. Butt, Ali Anwar
TL;DR
Cloud configuration and deployment are complex, and existing approaches may require extensive manual tuning. LADs combines LLM prompting, retrieval, few-shot examples, and feedback-based prompt chaining for configuration and maintenance. The authors report improved manual effort, resource use, and reliability, with feedback-based chaining resolving about 94% of configuration errors by the third attempt.
Problem
Cloud management requires substantial expertise and ongoing configuration work, while existing approaches can be system-specific and require manual tuning.
Source: §1 · pp. 1–2Method
LADs combines instruction prompting, RAG, few-shot learning, chain-of-thought, and feedback-based prompt chaining to generate configurations and refine them using deployment feedback.
Source: §1 · p. 2, §4.2 · p. 6Results
LADs reduces manual effort, optimizes resource utilization, and improves system reliability; feedback-based prompt chaining resolves approximately 94% of configuration errors by the third attempt.
Source: §6 · p. 8, §5.2.2 · p. 8Takeaways & Limitations
The benchmark and dataset provide a foundation for further research in automated DevOps validation.
Source: §1 · p. 2, §6 · p. 8The evaluation focuses on popular, seen distributed systems; extending LADs to unseen systems remains future work.
Source: Limitations · p. 9
Abstract
from arXiv · showhide
Automating cloud configuration and deployment remains a critical challenge due to evolving infrastructures, heterogeneous hardware, and fluctuating workloads. Existing solutions lack adaptability and require extensive manual tuning, leading to inefficiencies and misconfigurations. We introduce LADs, the first LLM-driven framework designed to tackle these challenges by ensuring robustness, adaptability, and efficiency in automated cloud management. Instead of merely applying existing techniques, LADs provides a principled approach to configuration optimization through in-depth analysis of what optimization works under which conditions. By leveraging Retrieval-Augmented Generation, Few-Shot Learning, Chain-of-Thought, and Feedback-Based Prompt Chaining, LADs generates accurate configurations and learns from deployment failures to iteratively refine system settings. Our findings reveal key insights into the trade-offs between performance, cost, and scalability, helping practitioners determine the right strategies for different deployment scenarios. For instance, we demonstrate how prompt chaining-based adaptive feedback loops enhance fault tolerance in multi-tenant environments and how structured log analysis with example shots improves configuration accuracy. Through extensive evaluations, LADs reduces manual effort, optimizes resource utilization, and improves system reliability. By open-sourcing LADs, we aim to drive further innovation in AI-powered DevOps automation.
1 Introduction
Cloud configuration is complex and costly to maintain, while existing approaches often fail to generalize across systems and dynamic workloads. LADs proposes an agentic LLM framework combining optimization techniques with feedback-based maintenance.
1 Introduction
- Cloud deployments demand broad expertise and continuous configuration updates, making maintenance time-consuming and expensive for startups.DevOps teams also spend substantial effort on observability, monitoring, and security, while many organizations struggle to understand cloud spending.Source: §1 · p. 1
- At large companies, configuration changes can outpace code updates, and misconfigurations remain a top cause of production incidents.The authors cite thousands of daily configuration-file diffs at Meta.Source: §1 · p. 1
- Existing configuration solutions are often system-specific, require extensive manual tuning, and leave large parameter spaces and changing workloads difficult to manage.RocksDB tuning can involve over 100 parameters, while validators may leave many configurations uncovered because of cost and complexity.Source: §1 · pp. 1–2
- LADs combines instruction prompting, hyperparameter tuning, few-shot learning, retrieval-augmented generation, and chain-of-thought for automated DevOps.Its feedback-based prompt chaining sends error logs and stage details back to the LLM agent to improve reliability.Source: §1 · p. 2
- LADs selectively combines domain knowledge and user intent, and introduces a benchmark and dataset for static and dynamic validation of automated DevOps.The framework is intended to reduce DevOps workload and misconfiguration risks while supporting cloud application management.Source: §1 · p. 2
2 Related Work
Prior LLM work addresses isolated configuration, debugging, or validation tasks, while LADs is presented as a holistic cloud-deployment framework combining adaptive optimization and structured feedback.
2 Related Work
- Earlier LLM approaches target narrower configuration tasks, including software-tool tuning, network configuration, and router configuration, with cited limitations in adaptability or error susceptibility.PreConfig is described as lacking adaptability to dynamic cloud environments, while router configuration approaches may require manual validation.Source: §2 · p. 2
- Prior debugging and validation work is characterized as reactive or static, whereas LADs uses RAG, feedback-based prompt chaining, and structured logs to refine configurations before deployment.The passage frames this approach as reducing the likelihood of failures rather than merely identifying them.Source: §2 · p. 2
- LADs combines adaptive optimization, structured feedback, and intelligent automation to address configuration correctness, resource efficiency, and deployment robustness.The paper contrasts this lifecycle-oriented approach with prior work focused on particular aspects of configuration, validation, or debugging.Source: §2 · pp. 2–3
3 Benchmark and Dataset
The paper introduces a human-curated benchmark and dataset for evaluating configuration generation against user intent and deployment behavior. It uses Python validators for static checks and Kubernetes deployments with official benchmarks for dynamic evaluation.
3 Benchmark and Dataset
- Static validation checks alignment with user intent, while dynamic validation deploys configurations and measures performance, cost, or scale using benchmark applications.The dynamic evaluation runs systems in multi-tenant Kubernetes environments and tracks completion time, cost, and resource consumption against user intents.Source: §3 · p. 3
- The dataset covers Dask, Redis, and Ray, selected for popularity and distinct microservice tasks, and compares their configuration complexity by parameter keys, nesting depth, and tested features.Redis has the highest Max Keys (221) and Avg Keys (77 ± 44), with 19 tested features; Ray has the deepest structure (Max Depth = 8, Avg Depth = 5.5 ± 0.68).Source: §3 · p. 3
- Dask's profile is described as balanced, with 192 keys and 16 tested features, making it relatively easy to configure.The complexity comparison includes key counts, nested-parameter depth, and tested features.Source: §3 · p. 3
- Each service has 60 dataset configurations across six difficulty levels, from beginner-friendly tasks to expert-level challenges.Prompts specify high-level tasks, while user intents give precise requirements.Source: §3 · p. 3
- Python-based validators check whether generated configurations align with prompts and user intent, providing more flexibility than regex-based validation.The paper notes that Python verifiers can also be generated by LLMs, simplifying future validator-generation automation.Source: §3 · p. 3
4 Method
LADs combines inference-phase optimizations to generate cloud configurations aligned with user intent while controlling token costs. Its methods address output structure and context provision, then use validation and feedback during deployment maintenance.
4.1 Structural integrity of configurations
- Because LLM outputs vary, LADs preprocesses responses and keeps temperature and top_p below 0.1 for more deterministic configuration generation.Preprocessing removes extraneous explanations, reasoning, or examples that could break YAML structure.Source: §4.1 · p. 4
- Instruction prompting, preprocessing, and randomness control make structural errors negligible, but prompt-alignment performance remains insufficient for fully automated DevOps pipelines.Source: §4.1 · p. 4
4.2 Context Provision
- LADs distinguishes structured context, such as configuration examples, from unstructured context, such as application documentation, user intent, and system data.The authors test RAG, few-shot learning, and Chain-of-Thought alongside instruction prompting, preprocessing, and randomness control.Source: §4.2 · pp. 4–5
- RAG yields only modest improvements over instruction prompting because chunking can fragment structured schemas and fail to retrieve essential configuration details.The authors therefore favor RAG for relevant chunks of unstructured context and few-shot examples for structured configuration schemas.Source: §4.2 · p. 5
- Few-shot examples provide structural references and lead to substantial accuracy gains by helping the model interpret configuration requirements and user intent.The examples include prompts, intents, and correct and incorrect outputs.Source: §4.2 · p. 5
- Chain-of-Thought produces only modest improvements because the model lacks domain-specific system specifications and configuration schemas.Source: §4.2 · p. 5
4 Method
- LADs combines instruction prompting, hyperparameter tuning, few-shot learning, Chain-of-Thought, RAG, and prompt chaining to target accuracy and efficiency while controlling token expenses.The framework treats LLMs as black-box modules for configuring a broad range of cloud applications.Source: §4 · p. 4, §4.2 · pp. 5–6
4.2 Context Provision
- For static validation, Chain-of-Thought checks output alignment with user intent, prompts, and shots; dynamic prompt chaining feeds filtered logs and stage-specific failures back to the model.Deployment errors, warnings, and details such as insufficient memory help narrow down root causes.Source: §4.2 · p. 6
5 Evaluation
LADs was evaluated through static checks of intent alignment, dynamic deployments, and configuration-generation overhead. Results show intent-dependent trade-offs in Redis deployments, gains from few-shot examples, and error correction through feedback-based prompt chaining.
5.1 Experimental Setup
- Static validation checked generated configurations against user intent, while dynamic validation deployed them in multi-tenant Kubernetes and monitored benchmark time, cost, and resource use.The authors describe dynamic validation as the first of its kind relative to related works that relied only on static validation.Source: §5.1 · pp. 6–7
5.2.1 Static Validation
- Few-Shot Learning was the most effective optimization for configuration accuracy, while Chain of Thought showed no noticeable improvement over vanilla models.The authors used Chain of Thought for validation when static human-curated validators were unavailable.Source: Table 2 · p. 6, §5.2.1 · p. 7
- Qwen2.5 14B achieved the highest accuracy across multiple applications and was selected for subsequent dynamic validation; newer, smaller models also outperformed older, larger models.The static comparison covered LLMs, applications, and optimization strategies.Source: Table 2 · p. 6, §5.2.1 · p. 7
5.2.2 Dynamic Validation
- “High Efficiency with balanced Scalability and Cost” achieved the lowest completion time for most Redis benchmarks, balancing costs between the low-cost and high-scalability intents.Low-cost intent used fewer resources and cost less but took longer; high-scalability intent used more resources and cost more.Source: §5.2.2 · pp. 7–8, Figure 3 · p. 7
- Approximately 94% of configuration errors were resolved by the third attempt through feedback-based prompt chaining.The method feeds errors and warnings back to improve configuration accuracy.Source: §5.2.2 · p. 8, Figure 4 · p. 8
5.3 Overhead of LADs
- 4–11 seconds on average were required for configuration generation per API call; the authors also report per-configuration costs in the order of a few ten-thousandths of a dollar.They characterize generation using cost-efficient LLMs under 20 billion parameters as rapid and affordable.Source: §5.3 · p. 8, Table 3 · p. 8, Table 4 · p. 8
6 Conclusion
LADs integrates multiple LLM techniques to automate cloud configuration and deployment, including maintenance based on real-time feedback. Evaluations report reduced manual effort, optimized resource use, and improved reliability.
6 Conclusion
- LADs combines IP, RAG, FSL, CoT, and feedback-based prompt chaining for cloud configuration and deployment, with evaluations reporting reduced manual effort, optimized resource use, and enhanced reliability.The authors also present their benchmark and dataset as a foundation for further research in automated DevOps validation.Source: §6 · p. 8
Limitations
LADs has a one-time manual onboarding cost for each new distributed system. Its current evaluation covers popular systems, with extending it to unseen systems left for future work.
Limitations
- Adding a distributed system requires manually designing example shots and supplying documentation and best practices.The authors identify automating this step with web search and agent assistance as future work.Source: Limitations · p. 9
- The evaluation focuses on popular, seen distributed systems; extending LADs to unseen systems remains future work.The authors say finding unseen systems that are also popular is difficult; extension would require generating documentation and example shots from code.Source: Limitations · p. 9
Ethical Considerations
The study reports that its dataset contains only user intents, prompts, and Python validator functions, and that its dynamic-validation benchmarks are publicly available. It cautions that LADs’ machine-learning outputs can have real-world consequences if used incorrectly.
Ethical Considerations
- The dataset contains user intents, prompts, and Python validator functions, with no assets or personally identifiable information collected.The authors report that they did not collect assets or personally identifiable information.Source: Ethical Considerations · p. 9
- Dynamic validation uses publicly available benchmarks from the distributed systems’ official documentation.The authors report that these benchmarks are public for evaluating the systems.Source: Ethical Considerations · p. 9
- LADs should be used carefully because incorrect use of its machine-learning outputs can have real-world consequences.Source: Ethical Considerations · p. 9
A.1 Examples of optimizations in LADs
This section illustrates prompting techniques used in LADs to improve performance and efficiency in distributed systems. Its examples cover Instruction Prompting and Few-Shot Learning for configuration generation.
A.1 Examples of optimizations in LADs
- LADs uses multiple prompting methodologies in its optimization examples, including Instruction Prompting and Few-Shot Learning.The section frames these examples as strategies for improving performance and efficiency in distributed systems.Source: §A.1 · p. 12
- Figure 6 demonstrates Few-Shot Learning for automated configuration generation in LADs.Source: §A.1 · p. 12
- Figure 5 showcases an example and the impact of Instruction Prompting in LADs.Source: Figure 5 · p. 12, §A.1 · p. 12
A.2 Static Validation with LADs
Static validation reports that LADs outperformed the other tested optimization techniques across the distributed systems, with Qwen2.5-Coder-14B performing best among the evaluated LLMs. Dynamic validation examines how Dask and Ray configurations align with user intents and workload trade-offs.
A.2 Static Validation with LADs
- Qwen2.5-Coder-14B was the best-performing evaluated model, with superior accuracy and efficiency across the benchmarks.Source: §A.2 · p. 12
- LADs outperformed the other tested optimization techniques across the evaluated distributed systems.The static validation compares accuracy across LLMs and optimization strategies for Dask, Redis, and Ray.Source: §A.2 · p. 12, Table 5 · p. 14, Table 6 · p. 15
A.3 Dynamic Validation with LADs
- Dask dynamic validation covers 15 benchmarks and shows configurations aligning with user intents, including trade-offs between scalability, efficiency, and cost.Some benchmarks are more efficient under High Scalability than under High Efficiency with balanced Scalability & Cost.Source: §A.3 · p. 13, Figure 7 · p. 17
- Ray dynamic validation covers four benchmarks, with reported improvements in resource utilization and cost-effectiveness as workloads vary.The figure presents completion times, processing costs, memory, CPU, and replica allocations.Source: §A.3 · p. 13, Figure 8 · p. 17