PaperlayerView exact revision on arXiv

Source-linked AI summary

Automated Synthesis of Cloud Emulators

Archit Bhatnagar, Zhenning Yang, Sarah McClure, Yiming Qiu, Sylvia Ratnasamy, Ang Chen

arXiv:2608.23842v1cs.SEcs.AIcs.DC

TL;DR

Testing DevOps programs against real cloud resources is costly, slow, and potentially unsafe, while manually building emulators cannot keep pace with cloud complexity and evolution. CloudEmu synthesizes emulator logic from documentation using neurosymbolic abstractions and aligns it with the live cloud, reducing VPC API errors in evaluation.

  • Problem

    Manual emulator construction requires tracking vast, evolving APIs and scattered resource interactions, limiting coverage and accuracy.

  • Method

    CloudEmu combines LLM-based documentation processing and code generation with state-machine abstractions, then tests and aligns the emulator against the actual cloud.

  • Results

    VPC signature mismatches fell from 29.9% to 9%, while response-format errors were completely fixed except for extra fields in some cases after applying state machines.

  • Takeaways & Limitations

    Cloud documentation can bootstrap emulator construction, while live-cloud alignment closes gaps involving incompletely documented initial state.

  • Takeaways & Limitations

    The emulator does not simulate invocation latency, rate limits, network latency, failures, concurrency, or consistency properties.

Abstract

from arXiv · show

DevOps programming (e.g., using CLI/API scripts or IaC frameworks) is key to cloud infrastructure management. Unlike traditional programming tasks, DevOps program testing needs provisioning and execution against actual cloud resources, which is often time-consuming, unsafe, and costly. Cloud emulators have gained popularity for easing DevOps program testing; they are generally API-level mocks that can execute DevOps programs in a local environment. Still, building these emulators remains challenging: developers must manually interpret extensive cloud documentation and handcraft logic for each service, API, and their interaction. This does not scale to the complexity of the cloud, which is further a moving target as the services and APIs evolve. CloudEmu is an automated approach that constructs emulators based on cloud documentation via neurosymbolic code synthesis. The key idea is to combine LLMs' general strengths in documentation understanding and code generation with cloud-specific symbolic abstractions that suppress hallucinations and enforce precision at scale, while using the real cloud as an oracle for automated testing, repair, and alignment. Our evaluation shows the effectiveness of CloudEmu on major cloud provider (AWS and GCP) services in both coverage and accuracy. CloudEmu outperforms the existing leading tool LocalStack, which was manually developed by a large team of engineers over a decade.

1 Introduction

CloudEmu automates cloud-emulator construction from provider documentation, addressing the cost, complexity, and dynamism that make manual development difficult. Its neurosymbolic workflow combines LLM inference with structured state-machine constraints and cloud-based alignment.

  • Motivation: Testing DevOps programs against real cloud resources is costly, time-consuming, and potentially unsafe, motivating local API-level emulators.Emulators expose matching interfaces and simulate execution without provisioning real resources.
  • Motivation: Manual emulator development requires interpreting extensive documentation and implementing each resource, API, and cross-resource interaction.Interdependent resources must be modeled together, such as VMs associated with subnets and VPCs.
  • Motivation: Cloud complexity and continual service and API evolution make manually maintaining reliable emulator coverage difficult at scale.Provider ecosystems contain many services and APIs and continue to change over time.
  • Approach: CloudEmu synthesizes emulation logic from cloud documentation and aligns the generated emulator with the live cloud.The approach aims to adapt to service changes and generalize across providers without multiplying engineering effort.
  • Approach: CloudEmu models resources as state machines whose hierarchical and dependency relationships constrain API-driven state evolution.The scaffold lets the LLM infer API behavior within preprogrammed structural constraints.
  • Approach: Documentation wrangling uses resource and API structure to retrieve and organize information more precisely than semantic indexing alone.The workflow combines symbolic parsing with LLM-based understanding of textual descriptions.
  • Approach: Alignment generates high-coverage traces, compares emulator and cloud effects, localizes divergence, and supports LLM-assisted repair.The goal is matching effects and, ideally, failure codes and messages for permissible and forbidden behaviors.
  • Results: CloudEmu’s evaluation reports high coverage and accuracy over leading manually built emulators such as LocalStack.The paper presents selective LLM use, symbolic constraints, and external-oracle feedback as broader system-building takeaways.

2 Motivation

Cloud emulators reduce the cost and risk of DevOps testing, but manual construction struggles with the cloud’s breadth, evolution, and behavioral complexity. CloudEmu addresses this through structured documentation processing, state-machine modeling, and cloud-based validation.

  • 2 Motivation: DevOps programs invoke cloud APIs to provision and update infrastructure, making thorough testing important but real-cloud testing costly.Emulators provide local API simulations that can expose infrastructure-program bugs before deployment.
  • 2.1 The Pitfalls of Manual Construction: LocalStack’s API coverage is under 40% even on popular cloud services.Table 1 characterizes incomplete coverage as a limitation of existing emulators.
  • 2.1 The Pitfalls of Manual Construction: Manual emulator construction struggles to keep pace with more than 240 AWS services, hundreds of APIs per service, and thousands of annual changes.This leads to coverage and accuracy limitations.
  • 2.1 The Pitfalls of Manual Construction: Across four services, LocalStack’s overall API coverage is approximately 35%, and one missing resource or API can prevent an entire DevOps program from executing.Network Firewall illustrates the gap by lacking several operations beyond CreateFirewall().
  • 2.1 The Pitfalls of Manual Construction: Manual implementations can diverge behaviorally from AWS, including accepting operations that AWS rejects with specific errors.Examples include deleting a VPC with an attached Internet Gateway and starting an already running instance.
  • 2.1 The Pitfalls of Manual Construction: Scaling manual development requires tracking documentation, resource state effects, API interactions, and tests across a changing stateful system.The paper illustrates this with subnet creation and VPC-deletion dependency constraints.
  • 2.2 CLOUDEMU: The Case for Automated Synthesis: Directly prompting an LLM to generate emulator code performs poorly because of context limitations, hallucinations, and imperfect documentation.CloudEmu therefore combines LLM capabilities with structured constraints and cloud-based alignment.
  • 2.2 CLOUDEMU: The Case for Automated Synthesis: Cloud documentation’s service/API structure and consistent resource descriptions enable symbolic parsing and targeted retrieval.This structure supports switching between symbolic parsing and neural understanding.

3 CLOUDEMU Design

CLOUDEMU combines documentation wrangling, neurosymbolic code synthesis, and cloud-oracle alignment to construct precise emulators from cloud documentation. Its resource-index and state-machine abstractions organize dependencies, constrain generated behavior, and support scalable testing and repair.

  • Documentation Wrangling: CLOUDEMU builds a resource index from cloud documentation, representing resources, dependencies, API parameters, and textual descriptions in a structured form.The index supports systematic traversal so emulation logic can be generated for all indexed resources and APIs.
  • Documentation Wrangling: The index addresses cloud-documentation scale by exploiting resource boundaries and cross-references rather than relying only on imprecise semantic retrieval.Documentation is decomposed by service, resource, and API boundaries, while dependencies guide navigation across related resources.
  • Code Synthesis: CLOUDEMU compiles resources into hierarchical state machines whose states, transitions, and dependency relationships constrain API-driven evolution.Symbolic scaffolding handles the general resource model, while generated code fills resource-specific behavior and interaction logic.
  • Code Synthesis: State inference symbolically captures externally visible resource state while neural inference extracts implicit runtime state from textual descriptions.For example, VpcId and CidrBlocks are maintained as explicit state because other resources or DevOps programs can observe them.
  • Code Synthesis: API signatures and response schemas are symbolically defined, while LLM-generated transition bodies implement the described behavior within the constrained scaffold.The index can specify typed inputs and serialized return schemas, reducing the amount of behavior the LLM must infer.
  • Automated Alignment: Testing must manage state-space explosion: 100 APIs, 5 parameter configurations, and length-3 traces yield approximately 125 million possibilities.The large space results from many APIs, parameter combinations, and interleavings among API calls.
  • Automated Alignment: Neurosymbolic test generation combines symbolic execution with LLM synthesis to target API branches and resource state, then uses cloud traces to repair emulator discrepancies.Alignment can seed undocumented defaults and correct formatting mismatches such as scalar-versus-list responses.

4 Evaluation

The evaluation examines CLOUDEMU’s emulation effectiveness, design contributions, synthesis and alignment overhead, and cross-cloud generalization. CLOUDEMU is implemented in 12.4k lines of Python, including components for documentation wrangling, scaffolding and synthesis, and automated alignment.

  • Evaluation Questions: The evaluation asks whether CLOUDEMU outperforms LocalStack, how its three designs affect performance, and how it generalizes across clouds.It also measures the overhead of synthesis and alignment.

alignment.

CloudEmu aligns documentation-synthesized emulators with real-cloud behavior through testing, discrepancy detection, and repair. Across AWS and GCP evaluations, its symbolic scaffolding and alignment improve coverage, fidelity, state consistency, and targeted discrepancy discovery.

  • Evaluation setup: CloudEmu executes CLI scripts and Terraform configurations through a local Flask gateway, enabling emulated DevOps execution.The EC2 and GCE emulators contain 88k and 65k lines of code, respectively.
  • Versus AWS: Against AWS traces, CLOUDEMU reaches 100% API coverage across selected EC2 cases, while LocalStack achieves around 70% case coverage.The comparison uses AWS as ground truth and includes 50 CLI and 50 Terraform cases.
  • Versus AWS: CLOUDEMU consistently outperforms LocalStack across fidelity levels, including Response Fidelity and State Consistency, with Terraform cases harder because they involve deeper API and state dependencies.Both systems match AWS error behavior on CLI cases, but LocalStack accepts invalid Terraform configurations and dependency violations.
  • Ablation study: The direct-to-code baseline reaches 66% coverage, whereas the Index, SM, and Full variants achieve perfect coverage.The resource index improves CLI Error Class accuracy by 11% and response fidelity by around 15.3%.
  • Ablation study: The state-machine scaffold yields a 15% Terraform Error Class gain and an 80% Response Fidelity gain by enforcing signatures, schemas, and dependency constraints.For CLI cases, the change is negligible because many operations lack checkable parameters.
  • Ablation study: Alignment reduces VPC signature mismatches from 29.9% to 9% and completely fixes response-format errors except for some extra fields.It also improves CLI response and state fidelity by correcting documentation-omitted defaults and incomplete state initialization.
  • Alignment efficiency: Coverage-guided cases expose more discrepancies than random scraped cases: Terraform discrepancy yield rises from 27% to 66%, a 2.5× improvement.At a budget of 50 cases per modality, coverage-guided cases exercise 79 unique EC2 API methods and include 37 absent from the scraped suite.

5 Discussion

CLOUDEMU’s design has limitations in performance and failure emulation, while provider-built emulators remain difficult because cloud implementations are hardware-coupled distributed systems. Its state-machine resource index also supports broader analysis of cross-cloud equivalence and cloud complexity.

  • Limitations: CLOUDEMU and LocalStack emulate API invocations but not invocation latency, rate limits, network latencies, concurrency, or consistency properties.Performance testing, chaos engineering, and semantic-property testing therefore require extensions beyond API fidelity.
  • Provider implementation: Cloud providers could derive emulators from actual implementations or documentation, but implementation reuse may require substantial reengineering because production systems couple hardware details with distributed execution.CLOUDEMU is particularly applicable to documentation-based emulator construction.
  • Broader uses: The resource index represents cloud complexity as a graph of interacting state machines, enabling metrics such as node counts and edge density.These metrics can support quantitative comparisons among cloud services.

6 Related Work

CLOUDEMU extends specification-mining and cloud-testing research by generating executable state-machine emulators from documentation and refining them with discrepancies against real clouds.

  • Specification mining: Prior specification-mining work extracts API specifications, assertions, temporal properties, or formal specifications, but generally stops at inferring invariants for existing implementations.CLOUDEMU instead generates executable emulation logic from cloud documentation.
  • Specification mining: Active automata learning interactively queries a black-box system to infer a state-machine model from traces or documentation.CLOUDEMU is positioned as the first system using LLMs to translate API documentation into a complete executable state-machine model for complex cloud services.
  • Cloud testing: Cloud-testing research commonly uses stateful fuzzing or differential testing to find implementation bugs, while related emulator studies identify behavioral gaps without principled repair.CLOUDEMU closes this loop by using discrepancies to refine its learned model.

7 Conclusion

CLOUDEMU addresses the costly, high-latency, and potentially unsafe testing of cloud DevOps programs by learning emulation logic from documentation and aligning it with cloud behavior. The evaluation demonstrates effectiveness against existing emulators and generalizability across AWS and GCP.

  • 7 Conclusion: CLOUDEMU uses a neurosymbolic approach to learn emulation logic from cloud documentation and automatically align the emulator with cloud behavior.The approach targets cloud APIs that are complex, evolving, and provider-specific.
  • 7 Conclusion: CLOUDEMU provides a local API-level alternative to direct cloud testing, which incurs cost and high latency and can create unwanted side effects when tests fail.This supports thorough validation before deployment for safety-critical cloud DevOps.
  • 7 Conclusion: CLOUDEMU is evaluated against existing emulators and shown to generalize across two major cloud providers.The conclusion identifies AWS and GCP as the evaluated clouds.
Loading 2608.23842v1…