PaperlayerView exact revision on arXiv

Source-linked AI summary

CloudEval-YAML: A Practical Benchmark for Cloud Configuration Generation

Yifei Xu, Yuning Chen, Xumiao Zhang, Xianshang Lin, Pan Hu, Yunfei Ma, Songwu Lu, Wan Du, Zhuoqing Mao, Ennan Zhai, Dennis Cai

arXiv:2401.06786v1cs.DCcs.AI

TL;DR

Cloud-native code generation lacks benchmarks, and model performance in cloud applications remains an open question. CloudEval-YAML builds a practical YAML benchmark with hand-written problems and tests, then evaluates LLMs and generation strategies, finding substantial performance differences and gains from multi-sample generation.

  • Problem

    Code-generation benchmarking for cloud-native applications is lacking, leaving the performance of LLMs on cloud applications an open question.

    Source: §1 · p. 1

  • Method

    CloudEval-YAML focuses on YAML and provides practical problems with reference files and unit tests, alongside an evaluation of 12 LLMs.

    Source: §1 · pp. 1–2

  • Results

    Proprietary models outperform open-source models by a large gap, multi-sample generation can bring remarkable gains, and few-shot prompting shows no significant gain.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The benchmark supports comparisons of LLMs for cloud configuration and identifies multi-sample generation as a way to improve performance and reduce cost.

    Source: §1 · p. 2

    Evaluation was constrained by PaLM API support for English only and a six-sample GPT-4 run due to API rate limits.

    Source: §4.2 · p. 8

Abstract

from arXiv · show

Among the thriving ecosystem of cloud computing and the proliferation of Large Language Model (LLM)-based code generation tools, there is a lack of benchmarking for code generation in cloud-native applications. In response to this need, we present CloudEval-YAML, a practical benchmark for cloud configuration generation. CloudEval-YAML tackles the diversity challenge by focusing on YAML, the de facto standard of numerous cloud-native tools. We develop the CloudEval-YAML benchmark with practicality in mind: the dataset consists of hand-written problems with unit tests targeting practical scenarios. We further enhanced the dataset to meet practical needs by rephrasing questions in a concise, abbreviated, and bilingual manner. The dataset consists of 1011 problems that take more than 1200 human hours to complete. To improve practicality during evaluation, we build a scalable evaluation platform for CloudEval-YAML that achieves a 20 times speedup over a single machine. To the best of our knowledge, the CloudEval-YAML dataset is the first hand-written dataset targeting cloud-native applications. We present an in-depth evaluation of 12 LLMs, leading to a deeper understanding of the problems and LLMs, as well as effective methods to improve task performance and reduce cost.

1 INTRODUCTION

CloudEval-YAML addresses the lack of cloud-native code-generation benchmarks by focusing on YAML and pairing practical problems with reference files and unit tests. Its evaluation examines 12 LLMs and reports substantial differences across model types and generation strategies.

1 INTRODUCTION

  • The benchmark addresses limited cloud-native code-generation evidence, where diverse application languages and interfaces and large-scale deployment complicate unified benchmarking and evaluation.The authors identify direct model comparisons as important for developing models and methods.Source: §1 · p. 1
  • CloudEval-YAML presents a first-of-its-kind YAML benchmark with 1,011 practical problems, reference YAML files, and unit tests, developed with more than 1,200 human hours.Problems include natural-language descriptions and optional code segments.Source: §1 · p. 1
  • Practical data augmentation rewrites questions using concise, abbreviated language and translations into cloud operation teams’ native languages.GPT-4 assists with rewriting, and each draft is manually reviewed.Source: §1 · pp. 1–2
  • A distributed evaluation platform completes the 1,011-problem evaluation in 30 minutes on 64 machines, versus 10 hours on one machine.The platform combines unit tests with shared Docker image caching.Source: §1 · p. 2
  • Proprietary models outperform open-source models by a large gap; multi-sample generation can bring remarkable gains, while few-shot prompting shows no significant gain.Dedicated code-generation models perform poorly on this task, and even the best-performing model makes simple mistakes.Source: §1 · p. 2
  • The authors evaluate 12 LLMs and test multi-sample generation, few-shot prompting, and models that predict unit-test results to reduce evaluation cost.Their contributions include a scalable automated evaluation platform and an in-depth benchmark analysis.Source: §1 · p. 2

2 THE CL O U DEV A L-YAML DATASET

The dataset combines practical YAML-generation problems with reference files and functional tests, then augments questions to better reflect cloud operators’ language. Its statistics describe broad application coverage and substantial construction effort.

2.1 Overall Structure

  • Each problem pairs a natural-language description, sometimes with YAML context, with a labeled reference YAML and automated tests for functional correctness.The reference supports text-level and YAML-aware comparisons; tests set up the environment, validate functionality, and clean up afterward.Source: §2.1 · p. 3
  • Problems are hand-picked from official documentation, StackOverflow, and ranked blogs to cover diverse applications and difficulty while avoiding ambiguous tasks and complex script generation.The authors write problem descriptions and unit tests, then label the YAML files.Source: §2.1 · p. 3

2.2 Practical Data Augmentation

  • Simplification reduces average word/token counts by 25.7%/20.9%, while translation targets users’ native languages; GPT-4 drafts are reviewed by experts and native speakers.A survey found that cloud operators use concise, abbreviated language and often prefer their native language.Source: §2.2 · p. 3, Table 1 · p. 3

2.3 Statistics of the CloudEval-YAML dataset

  • Creating CloudEval-YAML took more than 1,200 human hours; it covers Kubernetes functionalities and software including Envoy and Istio, with average solution length of 28.35 lines.That average is 4× HumanEval (6.3 lines) and MBPP (6.7 lines); solutions are capped at 531 tokens, and unit tests average 13.14 lines.Source: §2.3 · p. 4

3 THE CL O U DEV A L-YAML BENCHMARK PLATFORM

CloudEval-YAML’s platform combines YAML generation, score calculation, and cloud evaluation. It generates YAML from problem prompts, evaluates outputs with text, YAML-aware, and functional measures, and distributes unit testing across a scalable cluster.

3 THE CL O U DEV A L-YAML BENCHMARK PLATFORM

  • The platform comprises three major components: YAML generation, cloud evaluation, and score calculation.The implementation is written in 6.3k lines of Python and Bash.Source: §3 · p. 4

3.1 YAML Answer Generation

  • The query module combines a prompt template with each problem, generates YAML through a shared interface for local and remote models, and post-processes responses to extract clean YAML.Post-processing removes surrounding text or extracts code enclosed by supported delimiters.Source: §3.1 · pp. 4–5

3.2 Performance Score Calculation

  • Six metrics span text-level similarity, YAML-aware matching, and function-level unit tests to assess generated YAML from different perspectives.Text-level scores include BLEU, edit distance, and exact match; YAML-aware scores include key-value exact and wildcard match.Source: §3.2 · p. 5
  • Unit tests check functional correctness using expert-crafted scripts; Kubernetes-focused tests use Minikube, while Envoy applications are tested directly in Docker containers.Each test outputs 1 for success and 0 otherwise.Source: §3.2 · pp. 5–6

3.3 Cloud-based Evaluation Framework

  • A distributed master-worker cluster parallelizes unit tests, while a master-side Docker pull-through cache lets workers reuse images already downloaded by another worker.The master tracks test contexts, inputs, and outputs, and the cluster can scale dynamically.Source: §3.3 · p. 6
  • Over 20× faster: 64 workers with shared Docker image caching evaluate all 1011 problems in under 30 minutes versus over 10 hours on one machine.Parallel unit testing contributes 13× and shared caching 1.6× of the speed improvement.Source: §3.3 · p. 6

3.4 Running Cost of the Benchmark

  • $1.31 per run: the lowest-cost evaluated option pairs the GPT-3.5 API with a single GCP spot instance.The cost comparison covers two inference options and three cloud evaluation settings.Source: §3.4 · p. 6

4 EVALUATIONS ON CLOUDEVAL-YAML

The evaluation benchmarks 12 LLMs on CloudEval-YAML, analyzes problem types and failures, and tests multi-sample generation, few-shot prompting, and unit-test-score prediction.

4.1 Comprehensive Benchmark and Analysis

  • Proprietary models lead the best open-source model by 4.84× on unit test score, while answer length and application category affect performance.Scores drop steeply for medium-long answers; Envoy questions are harder, while code context usually has little influence.Source: §4.1 · p. 7

Performance analysis of the practically augmented

  • Simplification reduced unit-test passes more for small models than large ones; translation sharply affected WizardCoder and Llama-2-7b-chat but not Llama-2-70b-chat.The reported pass counts changed from 179 to 164 for GPT-4, 26 to 17 for Llama-2-13b-chat, 24 to 2 for WizardCoder-34b-v1.0, 13 to 5 for Llama-2-7b-chat, and 30 to 32 for Llama-2-70b-chat.Source: Performance analysis of the… · p. 7, Table 5 · p. 9
  • GPT-4 makes more simple Category 1 errors than Llama2-7B/70B, while both Llama2 models often produce valid YAML with the correct kind that still fails unit tests.The six categories range from empty or very short answers to YAML that passes the unit test.Source: Performance analysis of the… · p. 7

4.1 Comprehensive Benchmark and Analysis

  • GPT-4 leads all text-level, YAML-aware, and function-level scores, with a unit test score of 0.515; Llama-2-70b-chat leads open-source models except on exact_match.The benchmark compares average zero-shot scores across models.Source: §4.1 · p. 7, Table 4 · p. 8

4.2 Multi-sample Generation

  • With pass@k counting a problem as passed if any of k samples passes, 20-sample generation raises unit test scores for Llama-2-70B, PaLM-2, and GPT-3.5 by 30%, 37%, and 39%.GPT-3.5 with six samples can beat GPT-4 with one, despite a reported 30× cost difference; PaLM-2 reaches GPT-3.5's level after five samples.Source: §4.2 · pp. 8–9, Figure 8 · p. 8

4.3 Few-shot Prompting

  • Few-shot prompting does not significantly improve unit-test scores across model sizes; the authors suggest three examples may cover too few cloud configuration types.The prompt used three example question-answer pairs.Source: §4.3 · p. 9

4.4 Predicting Unit Test Results

  • A classifier trained on text-level and YAML-aware scores roughly preserves model rankings, but relative error can reach 80%, so unit tests remain necessary for accurate evaluation.The evaluation masks one model's unit scores and trains on the other 11 models.Source: §4.4 · p. 9, Figure 9 · p. 9
  • Key-value wildcard match is the most important and accurate prediction feature, while BLEU and Edit distance are important but noisy metrics.The feature analysis uses SHAP values.Source: §4.4 · pp. 9–10, Figure 9 · p. 9

5 RELATED WORKS

Related benchmarks cover general code generation, domain-specific tasks, and multilingual prompting, but the cited alternatives do not target cloud applications as realistic hand-written problems.

5 RELATED WORKS

  • HumanEval and MBPP contain hand-written programming problems, and WikiSQL is hand-annotated, but none targets cloud applications.HumanEval covers Python programming, algorithms, and simple mathematics; MBPP contains entry-level Python problems; WikiSQL maps natural language queries to SQL.Source: §5 · p. 10
  • DS-1000 and Ansible-YAML inform CloudEval-YAML's design, but CloudEval-YAML focuses on hand-written, realistic problems from real cloud applications.DS-1000 adapts Python data-science problems from StackOverflow; Ansible-YAML supports natural-language-to-Ansible-YAML generation.Source: §5 · p. 10
  • CodeApex and MCoNaLa support multilingual prompting, while Lyra pairs Python with embedded SQL and English and Chinese comments.The cited datasets provide multilingual or bilingual code-generation settings.Source: §5 · p. 10

6 CONCLUSION

CloudEval-YAML combines a hand-written benchmark for realistic cloud YAML problems with an end-to-end evaluation platform. The appendix reports YAML use across leading cloud-native applications and gives the prompt template’s output instructions.

6 CONCLUSION

  • CloudEval-YAML is the first hand-written dataset of more than 1000 realistic problems for YAML configuration generation in cloud applications, paired with an end-to-end evaluation platform and functional-correctness evaluation.Source: §6 · p. 10

A YAML STATISTICS

  • 90 of the top 100 most-starred cloud-native applications contain more than 10 YAML files.The survey also notes that leading tools such as GitLab, Kubernetes, and Elastic depend heavily on YAML.Source: A YAML STATISTICS · p. 13

B PROMPT TEMPLATE

  • The prompt template requests complete YAML only, without descriptions or Markdown, and asks models to make a logical choice when details are missing.It also says not to ask for more details.Source: B PROMPT TEMPLATE · p. 13

C.1 Sample #1

Sample #1 presents a natural-language request to create a Kubernetes DaemonSet configuration, then checks the generated configuration through deployment and unit-test commands.

C.1 Sample #1

  • The sample asks for a DaemonSet using the latest nginx image, a registry service on port 80 with hostPort 5000, specified environment variables, and CPU and memory requests.The prompt is natural-language-only, representing creation of a new configuration.Source: §C.1 · p. 14
  • The test script applies the YAML and waits for the selected pod to become ready before checking the exposed service.It checks HTTP status, required environment variables, and CPU and memory limits.Source: §C.1 · pp. 14–15
  • The sample reports unit-test success only when the passed-test count equals the total-test count.The requested container resources are requests, while the displayed test checks limits.Source: §C.1 · pp. 14–15

C.2 Sample #2

Sample #2 asks for a browser-accessible load-balancer service for an existing nginx deployment. Its checks apply the YAML and invoke the service through Minikube.

C.2 Sample #2

  • The sample provides Deployment YAML and asks for a service with a load balancer, an nginx selector, and port 80 that is accessible through a browser.The provided context is intended for extending existing functionality.Source: §C.2 · p. 15
  • The test applies the generated YAML, waits for deployments to become ready, and runs the Minikube service command, checking its output for the expected service-opening message.The command output is captured and searched for that message.Source: §C.2 · p. 16

C.3 Sample #3

Sample #3 frames a StackOverflow-sourced YAML debugging task around an invalid Kubernetes Ingress and its server error. The test applies the repaired YAML and checks Ingress synchronization and expected service details.

C.3 Sample #3

  • The debugging prompt supplies an invalid Ingress YAML and asks for a complete corrected file after presenting the server error.The query is sourced from StackOverflow and includes the raw YAML and error report.Source: §C.3 · pp. 16–17
  • The test applies the YAML, waits for Ingress synchronization, and checks that the Ingress description contains “test-app:5000.”The displayed error identifies unknown fields in annotations and the Ingress backend service fields.Source: §C.3 · p. 17

D PROMPTS FOR DATA AUGMENTATION

The appendix gives prompts and examples for simplifying cloud-configuration questions and translating them into fluent Chinese. Both transformations preserve the information needed to request complete YAML, including code where relevant.

D.1 Prompt for question simplification

  • Question simplification asks for concise wording with abbreviations while retaining necessary information and leaving code unchanged.The prompt says not to answer the question directly and provides examples before the input to rewrite.Source: §D.1 · p. 18
  • Other simplification examples retain LimitRange CPU and memory settings and MySQL Service, Deployment, storage, and naming requirements.The MySQL example includes port 3306, MYSQL_ROOT_PASSWORD, a volume mount, and a matching PersistentVolumeClaim.Source: §D.1 · p. 18
  • A Kubernetes Secret example condenses a request to source pod environment variables from a file while preserving the full-YAML requirement.The example assumes a Secret named mysql-secret containing all values.Source: §D.1 · p. 19
  • An Istio example compresses a DestinationRule request while retaining the ratings service, production namespace, and LEAST_REQUEST strategy.A second example retains the testversion subset, v3 labels, and ROUND_ROBIN strategy.Source: §D.1 · p. 19

D.2 Prompt for question translation

  • Question translation asks for fluent Chinese in a developer's tone, without answering the question, and leaves code unchanged.The translated question need not be formal or a complete sentence.Source: §D.2 · p. 20
  • The translation examples include a Kubernetes LimitRange request and a MySQL Service-and-Deployment request with configuration details.The MySQL source specifies port 3306, an environment variable, a volume mount, a matching claim, and names and labels.Source: §D.2 · p. 20
  • The Kubernetes Secret example is translated into Chinese while retaining its environment-variable question, pod context, and full-YAML request.It assumes the named mysql-secret contains all values.Source: §D.2 · p. 21
  • The Istio translation preserves the ratings service's LEAST_REQUEST strategy and the testversion subset's v3 label and ROUND_ROBIN strategy.The examples place the service in the bookinfo application's production namespace.Source: §D.2 · pp. 21–22

E PERFORMANCE ANALYSIS ON DIFFERENT AFFECTING FACTORS

The passage identifies a table presenting performance analysis on different affecting factors.

E PERFORMANCE ANALYSIS ON DIFFERENT AFFECTING FACTORS

  • Table 9 is titled as a performance analysis on different affecting factors.The passage provides the table title but no results or factor comparisons.Source: Table 9 · p. 22
Loading 2401.06786v1…