PaperlayerView exact revision on arXiv

Source-linked AI summary

IaC-Guard-V: A Verification Framework for LLM-Generated Infrastructure-as-Code Repairs

Lokesh Chauhan

arXiv:2609.28488v1cs.SE

TL;DR

IaC repair lacks a direct trustworthiness check: syntactically plausible changes may fail to resolve the target issue or introduce regressions. IaC-Guard-V evaluates repairs through multiple verification dimensions and tests three repair strategies across a Terraform and Kubernetes benchmark. Verification-guided iteration improves full-verification rates over single-shot repair.

  • Problem

    The trustworthiness of LLM-generated IaC repairs is underexplored, while scanner disagreement, provider-specific semantics, and absent executable test suites complicate verification.

    Source: §1 · pp. 1–2

  • Method

    IaC-Guard-V evaluates repairs with syntax, target-issue resolution, regression-safety, and patch-minimality checks across a benchmark of Terraform and Kubernetes artifacts.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    Verification-guided iterative repair achieves 68–92% verified-fix rates, compared with 32–50% of single-shot repairs passing full verification, despite 100% syntactic validity across models.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Llama 4 Maverick with verification-guided repair reaches 70% VFR at $0.0024 per verified fix, versus Opus plain at 46% VFR and $0.0287 per verified fix.

    Source: §7 · p. 9

    Because the benchmark comes from Checkov’s test suite rather than production repositories, it may not capture the full complexity of production IaC codebases.

    Source: §8 · p. 10

Abstract

from arXiv · show

Infrastructure-as-Code (IaC) misconfigurations are a leading cause of cloud security incidents, and Large Language Models (LLMs) are increasingly proposed as automated repair agents. Yet the trustworthiness of LLM-generated IaC repairs remains underexplored. IaC presents distinctive verification challenges: security scanners may behave differently across tools and configurations, infrastructure semantics cannot be validated through ordinary unit tests, and provider-specific rules create a fragmented verification landscape. We present IaC-Guard-V, a verification-centered framework that evaluates AI-generated IaC repairs through four dimensions: syntactic validity, target-issue resolution, regression safety, and patch minimality. We construct a benchmark of 70 real-world misconfigured Terraform and Kubernetes artifacts spanning 70 unique scanner rules and eight violation classes, and evaluate three repair strategies across three LLM families in 630 runs. Although all models achieve 100% syntactic validity, only 32-50% of single-shot repairs pass full verification. Verification-guided iterative repair significantly improves verified-fix rates to 68-92%. An open-source model with verification-guided repair outperforms the strongest commercial model without verification at one-twelfth the cost per verified fix. Kubernetes repairs reach near-perfect rates for commercial models but require verification-guided iteration for the open-source model. Structured prompting consistently reduces Terraform repair quality, challenging common assumptions about constrained LLM output. The benchmark and artifacts are released for reproducibility.

1. INTRODUCTION

IaC misconfigurations are difficult to repair safely because syntactic plausibility does not establish that a fix resolves the target issue without regressions. IaC-Guard-V evaluates repairs across verification gates and shows that verification-guided iteration improves full-verification rates.

1. INTRODUCTION

  • A motivating Terraform repair changed the wrong attribute despite being syntactically valid; verification caught the issue, and a second attempt produced the correct fix.The example concerns an EBS encryption finding and illustrates target-issue resolution as a distinct check.Source: §1 · p. 1, Figure 1 · p. 2
  • IaC repairs are difficult to trust because scanners can disagree, provider-specific semantics vary, and fixing one resource can introduce regressions in others.IaC artifacts lack executable test suites, so ordinary unit-test verification does not transfer directly.Source: §1 · pp. 1–2
  • IaC-Guard-V evaluates syntactic validity, target-issue resolution, and regression safety as binary gates, while measuring patch minimality separately.Its benchmark contains 70 real-world artifacts spanning 70 unique scanner rules and eight violation classes.Source: §1 · p. 2
  • All models achieve 100% syntactic validity, but only 32–50% of repairs pass full verification under single-shot prompting; verification-guided iterative repair achieves 68–92% VFR, statistically significantly outperforming baselines.Source: §1 · p. 2

2. BACKGROUND AND RELATED WORK

The related-work review situates IaC-Guard-V among LLM-based program repair, IaC repair and generation, and IaC benchmarks. It identifies a gap in combining LLM repair, multidimensional verification, and cross-technology evaluation.

2. BACKGROUND AND RELATED WORK

  • The paper organizes related work into three areas: LLM-based automated program repair, IaC-specific repair and generation, and IaC benchmarks and defect taxonomies.Table I positions the study against closely related work.Source: §2 · p. 2

2.1 LLM-Based Automated Program Repair

  • Prior LLM-based automated program repair includes prompting, fine-tuning, agentic systems, and iterative repair using test feedback.The review describes DRV as a related iterative verification paradigm, while noting its reliance on executable-code tests.Source: §2.1 · pp. 2–3

2.2 IaC-Specific Repair and Generation

  • Prior IaC work spans generation, constraint-based repair, and LLM repair, but the reviewed systems do not combine cross-technology evaluation with multidimensional verification of LLM repairs.The cited systems vary in technology scope and verification focus.Source: §2.2 · p. 3

2.3 IaC Benchmarks and Defect Taxonomies

  • Earlier IaC benchmarks and taxonomies characterize security smells, defects, and code-generation capability across tools and technologies.The review states that no prior work combines LLM-based repair, syntax, resolution, regression, and minimality verification, and cross-technology IaC evaluation in one framework.Source: §2.3 · p. 3

3. IAC-GUARD-V FRAMEWORK

IaC-Guard-V evaluates LLM-generated repairs with three binary verification gates and a separate patch-minimality measure. Its strategies range from single-shot prompts to feedback-guided retries.

3.1 Verification Gates

  • A repair counts as a verified fix only if it passes syntactic validity, target-issue resolution, and regression-safety checks.Patch minimality is measured separately because the paper sets no universal acceptable patch-size threshold.Source: §3.1 · pp. 3–4

3. IAC-GUARD-V FRAMEWORK

  • The pipeline sends an artifact and scanner finding to an LLM repair strategy, then evaluates the result across four gates.In verify-loop mode, failed verification results are returned to the model for up to two retries.Source: §3 · p. 3, Figure 2 · p. 4

3.2 Verified-Fix Rate

  • Verified-fix rate counts repairs that pass all three binary gates, divided by the number of benchmark items.The metric is more conservative than counting syntactically valid or plausible-looking repairs.Source: §3.2 · p. 4

3.3 Repair Strategies

  • The three strategies are plain prompting, structured prompting with Checkov context and JSON output, and verification-guided repair that feeds failed-gate results back to the model.The verification-guided strategy allows up to two retries after its first attempt.Source: §3.3 · p. 4

3.4 Implementation

  • The harness runs Checkov, checks whether the target rule remains, compares failed-rule sets for regressions, and measures patch size using a unified diff.The implementation also records intermediate verification results, token counts, and latency.Source: §3.4 · p. 5

4. BENCHMARK CONSTRUCTION

The benchmark draws Terraform and Kubernetes artifacts from Checkov’s official test suite and selects examples across rules and violation types. Baseline scanner outputs support target-finding checks and regression detection.

4.1 Source and Selection

  • The benchmark contains 70 artifacts: 50 Terraform and 20 Kubernetes, selected from Checkov’s official test suite using one item per unique rule and oversampling of minority Terraform violation classes.The source suite contains 369 unique Terraform rules and 215 unique Kubernetes rules.Source: §4.1 · p. 5

4.2 Violation Taxonomy

  • The benchmark classifies its items into eight violation classes based on security impact.Table II summarizes the benchmark composition.Source: §4.2 · p. 5, Table II · p. 5

4.3 Baseline Establishment

  • For each item, baseline Checkov output confirms the target rule is present and supplies the reference set used to detect regressions.The target rule was confirmed for all 70 items; baseline outputs and the selection manifest are included in the replication package.Source: §4.3 · p. 5

5. EXPERIMENTAL SETUP

The experiments compare three models and three repair strategies on Terraform and Kubernetes artifacts, using a reproducible protocol and statistical tests for Terraform method differences.

5.1 Models

  • Three models span two commercial Claude systems and openweight Llama 4 Maverick, enabling comparisons across capability and openness.Opus and Sonnet are the commercial models; Maverick has 17B active parameters in a mixture-of-experts architecture.Source: §5.1 · p. 5

5.2 Configuration

  • All models run at temperature 0; verify-loop allows two retries, and all verification uses Checkov v3.2.517.Maximum output length is 4,096 tokens; prompts are in the replication package.Source: §5.2 · p. 5

5.3 Evaluation Protocol

  • Each of 70 benchmark items is tested with all three strategies on all three models, producing 630 runs.The protocol records responses, extracted patches, four verification results, token counts, latency, and per-attempt results for verify-loop.Source: §5.3 · p. 5

5.4 Statistical Methods

  • Terraform VFRs use 95% bootstrap intervals and pre-specified Cochran’s Q and corrected McNemar tests; Kubernetes comparisons are descriptive because commercial models reach 100%.Pairwise Terraform tests use Bonferroni correction at α = 0.0083; Cliff’s delta is reported as effect size.Source: §5.4 · p. 6

6. RESULTS

Across 630 runs, verification-guided repair improves Terraform results over single-shot prompting, with outcomes varying by violation class, technology, model, cost, and patch minimality.

6.1 RQ1: Effectiveness of Verification-Guided Repair

  • 92% VFR (Opus), 70% (Maverick), and 68% (Sonnet) with verify-loop, versus 32–50% for single-shot methods; verify-loop beats both baselines across models.All models achieve 100% syntactic validity, but this alone does not ensure target-issue resolution or regression safety.Source: Figure 3 · p. 6, §6.1 · p. 6, Table III · p. 6

6.2 RQ2: Variation by Violation Class

  • 47–97% verify-loop VFR across violation classes; missing-encryption is easiest with single-attribute fixes, while public-exposure gains most over plain (94% vs. 17%).Insecure-defaults and “other” are hardest; Opus reaches 100% on insecure defaults while Sonnet and Maverick reach 20%.Source: §6.2 · pp. 6–7, Table IV · p. 7, Table V · p. 7

6.3 RQ3: Cross-Technology Generalization

  • Commercial models achieve 100% Kubernetes VFR across methods, while Maverick rises from 45% with plain prompting to 100% with verify-loop.Structured prompting raises Maverick to 95% on Kubernetes, unlike its quality-degrading effect on Terraform.Source: §6.3 · p. 7, Table VI · p. 7

6.4 RQ4: Cost-Effectiveness

  • 78.6% VFR at $0.002 per verified fix (Maverick verify-loop) compares with 61.4% at $0.029 (Opus without verification), nearly 12× higher cost.Maverick verify-loop averages 5.0s latency; Sonnet verify-loop costs $0.044 per fix and averages 17.6s.Source: §6.4 · p. 7, Table VII · p. 7

6.6 RQ5: Failure Analysis

  • 30 of 50 Terraform items were fixed by all models, four by none, and 16 had model-dependent outcomes.The 35 verify-loop failures included harness isolation limits, one unresolved graph-based rule, and model failures; a third retry added only 2 percentage points on average. Sonnet retry confusion occurred in 3/50 items, while 30 items had full cross-model agreement and seven were fixed exclusively by Opus.Source: §6.6 · p. 8, Figure 4 · p. 8

7. DISCUSSION

The discussion finds verification-guided repair can matter more than model choice, while prompting effects vary by IaC technology. Structured prompting trades Terraform correctness for smaller patches, but improves Maverick’s Kubernetes results; verify-loop gains also diminish after two retries.

7. DISCUSSION

  • Maverick verify-loop reaches 70% VFR at about 8% of Opus plain’s cost per verified fix, while Opus verify-loop reaches 92% VFR versus 46% for Opus plain.The authors argue that these differences suggest verification infrastructure can yield greater returns than upgrading model capability.Source: §7 · p. 9
  • On Terraform, structured prompting underperforms plain prompting (32–42% vs. 44–50%) and introduces regressions in 4–10% of cases.The authors attribute this to serialization-related escaping errors or truncation, and to prompts that focus narrowly on one resource and miss other instances of the rule.Source: §7 · p. 9
  • Structured prompting yields the smallest Terraform patches (median diff ratio 0.067–0.086), but can miss necessary changes elsewhere in the file.The authors describe a correctness–minimality tradeoff: plain prompting makes broader changes more likely to resolve the target issue, though patches may be larger.Source: §7 · p. 9
  • On Kubernetes, structured prompting raises Maverick’s performance from 45% to 95%, a reversal the authors attribute to manifests’ simpler, single-workload structure.With no other resources to miss, the narrow focus is advantageous rather than a liability.Source: §7 · p. 9
  • Most verify-loop fixes occur on attempts 1–2, and a third attempt adds only 2 percentage points on average.Some Sonnet items show retry confusion, where feedback after a failed first attempt leads to overcorrection, although the net effect remains positive (+12 items).Source: §7 · p. 9
  • The authors recommend plain prompting with post-hoc verification for commercial-model Kubernetes repairs, iterative verification for Terraform or open-source models, and two retries as the cost-effective sweet spot.They recommend a CI/CD verification gate and identify multi-scanner consensus as a next step; the current framework relies on one scanner.Source: §7 · p. 9

8. THREATS TO VALIDITY

The authors identify scanner, measurement, and benchmark-scope constraints that qualify the evaluation. Their reported statistical comparisons are mostly significant under the corrected threshold, with one comparison narrowly missing it.

8. THREATS TO VALIDITY

  • Checkov is the sole scanner; two items produce V2 false negatives, and one graph-based rule remains unresolved by every model.The authors also note that results may depend on prompt wording because they use one template per strategy.Source: §8 · p. 9
  • VFR treats V1–V3 as binary gates and leaves V4 minimality informational, so even a whole-file rewrite can pass if otherwise correct.The authors say human evaluation of repair-intent preservation would complement the automated metrics.Source: §8 · pp. 9–10
  • The 70-item benchmark comes from Checkov’s test suite rather than production repositories and may not capture production IaC complexity; results also reflect model versions evaluated in April 2026.The authors plan to extend the benchmark to additional IaC technologies, production-repository items, and multi-scanner verification.Source: §8 · p. 10
  • Five of six pre-specified comparisons meet the Bonferroni-corrected threshold (α = 0.0083); the sixth narrowly misses it but is significant under Holm-Bonferroni.All six are significant at uncorrected α = 0.05.Source: §8 · p. 10

9. CONCLUSION

IaC-Guard-V evaluates LLM-generated IaC repairs with verification gates and patch minimality. Single-shot syntax validity does not ensure verified repairs, while verification-guided iteration improves verified-fix rates.

9. CONCLUSION

  • 100% of repairs achieve syntax validity, but only 32–50% of single-shot repairs pass full verification; verification-guided iteration raises verified-fix rates to 68–92%.Opus verify-loop reaches 92% (95% CI: 84–98%).Source: §9 · p. 10
  • IaC-Guard-V evaluates repairs through three binary gates—syntactic validity, target-issue resolution, and regression safety—and the informational metric of patch minimality.The benchmark includes 70 Terraform and Kubernetes artifacts, 70 unique scanner rules, and eight violation classes, evaluated in 630 runs.Source: §9 · p. 10
  • Future work targets multi-scanner consensus, human evaluation of repair-intent preservation, and broader technology and production-repository benchmarks.The proposed technology extensions include CloudFormation, Pulumi, and Ansible.Source: §9 · p. 10
Loading 2609.28488v1…