PaperlayerView exact revision on arXiv

Source-linked AI summary

Compared to What? A Human-Anchored Security Benchmark for LLM-Generated Infrastructure-as-Code

Animesh Shaw

arXiv:2608.28021v1cs.CRcs.AIcs.MAcs.SE

TL;DR

Prior evaluations lacked a human reference measured through the same toolchain, limiting interpretation of model vulnerability counts. This paper supplies a size-matched human baseline and decomposes reasoning modes, finding consistently higher model vulnerability density and only modest benefits from vendor extended thinking.

  • Problem

    Without a human reference measured through the same toolchain, vulnerability counts cannot distinguish model carelessness from larger artifacts or scanner effects.

  • Method

    The paper scans human-authored and model-generated IaC through the same three-engine toolchain, matches comparisons by resource count, and separates standard generation, prompted chain-of-thought, and vendor extended-thinking APIs.

  • Results

    Models produce approximately 3.2×–3.9× the human vulnerability density across vendors; vendor reasoning modestly improves security over prompted chain-of-thought, while prompted chain-of-thought provides no measurable benefit.

  • Takeaways & Limitations

    Security comparisons should use size-matched human baselines, and reasoning effects should be interpreted alongside measured reasoning-token expenditure.

  • Takeaways & Limitations

    The human corpus is not a matched control because its templates were not written against the benchmark scenarios and many are curated examples rather than production infrastructure.

Abstract

from arXiv · show

Large language models are increasingly used to author Infrastructure-as-Code (IaC), where a single insecure default can be deployed directly into production. Prior evaluations report raw vulnerability counts for model-generated IaC, but without a human baseline they cannot determine whether models are actually worse than engineers. We introduce GenIaC-SecBench, a benchmark of 100 deployment scenarios stratified by architectural complexity, evaluated across 12 model configurations from four vendors, producing 1,196 IaC artifacts scanned by three independent policy engines (Checkov, Trivy, KICS). Critically, we also scan 634 human-authored IaC templates with the same toolchain, providing the first size-matched human security baseline. Vulnerability density is strongly inverse to artifact size (Spearman $ρ= -0.55$, $p < 10^{-77}$), meaning unmatched comparisons measure size rather than security. When matched on declared-resource count, all model configurations fall within 3.21x--3.87x the human vulnerability density, with the gap widening for simpler tasks (4.9x at one resource, 1.4x at twenty or more). We decompose reasoning into standard generation, prompt-engineered chain-of-thought, and vendor extended-thinking APIs. Vendor extended thinking significantly outperforms prompted chain-of-thought ($-12.0\%$, $p = 0.0013$), while prompted chain-of-thought is indistinguishable from standard generation ($-1.3\%$, n.s.). Token instrumentation shows extended thinking uses under 1\% of the output budget, explaining its bounded effect. Two negative results also emerge: deployability does not correlate with vulnerability ($r = 0.158$, $p = 0.625$), and classical complete-case Friedman testing is infeasible for realistic benchmark designs, motivating the Skillings-Mack statistic. All code, data, and regeneration scripts are released.

I. INTRODUCTION

IaC misconfigurations are provisioned as written, making security evaluation consequential as LLMs increasingly generate infrastructure code. This paper addresses the missing human reference needed to interpret model vulnerability counts.

  • I. INTRODUCTION: IaC misconfigurations are deployed as written, so insecure defaults can directly expose provisioned infrastructure.The paper distinguishes IaC security from latent application-code defects because templates declare deployed resources.
  • I. INTRODUCTION: Prior IaC evaluations report model vulnerability counts without a same-toolchain human baseline.Without that anchor, higher counts could reflect carelessness, larger artifacts, or scanner behavior rather than security differences.
  • I. INTRODUCTION: The benchmark contributes a size-matched baseline from 634 human-authored templates scanned with the same three engines.It also stratifies comparisons by declared-resource count and separates prompted chain-of-thought from vendor extended thinking.
  • I. INTRODUCTION: IaC security differs from application-code security because the analyzed unit is a declarative resource graph, with defaults rather than control flow determining risk.This motivates infrastructure-specific evaluation rather than direct reuse of function-level application-code benchmarks.
  • I. INTRODUCTION: The study separates prompt-based and API-based reasoning because they change the prompt and allocated computation through distinct mechanisms.The paper notes that conflating these mechanisms is easy and has not been addressed in prior security evaluations.

D. LLM-as-a-judge

The benchmark uses controlled, stateless generation across diverse deployment scenarios and model configurations, while recording protocol details that can affect security posture. Outputs are validated, token-instrumented, and generated without security-specific instructions.

  • D. LLM-as-a-judge: The benchmark covers 100 deployment scenarios across four cloud or infrastructure formats, split into 60 simple and 40 complex tasks.Scenarios specify functional requirements only and span AWS, Azure, GCP, and provider-agnostic Kubernetes.
  • D. LLM-as-a-judge: Twelve configurations include nine models across four vendors plus standard, prompted-CoT, and vendor-thinking variants on one base model.The three Claude Opus 4.6 rows differ only in reasoning mode, isolating that variable.
  • D. LLM-as-a-judge: Each request is a single stateless API call with no conversation history, retrieval, few-shot examples, or shared memory.The fixed system prompt requests production-ready code in one fenced block, while the scenario supplies only target-format functional instructions.
  • D. LLM-as-a-judge: No prompt mentions security, hardening, compliance, or specific controls, so observed security posture reflects model defaults rather than security instruction-following.Prompted chain-of-thought adds only a step-by-step reasoning instruction before the code block.
  • D. LLM-as-a-judge: Four complex scenarios were missing because generation reached the 128k-token maximum, while truncated responses were discarded.Per-request token usage, including reasoning tokens, was logged for later analysis.

D. Validation and scanning

Generated IaC is schema-validated and scanned comprehensively by three independent policy engines, while a same-toolchain human corpus supplies the comparison baseline. Vulnerability density is computed per declared resource and analyzed with methods suited to incomplete benchmark blocks.

  • D. Validation and scanning: 1,196 of 1,196 artifacts were scanned by Checkov, Trivy, and KICS.A machine-readable coverage manifest was regenerated on every run, supporting complete multi-engine coverage.
  • D. Validation and scanning: The human corpus contains 634 IaC templates from three public repositories scanned with identical tools and configuration.Applying the same ruleset to both corpora makes ruleset bias apply equally to the two sides.
  • D. Validation and scanning: Vulnerability density divides findings from all three engines by declared resources parsed from each artifact’s syntax tree.Resource counts come from AST parsing rather than scanner output.
  • D. Validation and scanning: Skillings–Mack tests model differences within complexity strata, with Wilcoxon-Holm post-hoc comparisons and negative-binomial GEE counts using resource count as exposure.The method accommodates incomplete blocks while estimating per-resource incidence.
  • D. Validation and scanning: Generation produced 1,196 of 1,200 possible artifacts, and the scanners reported 38,803 findings.Findings totaled 14,017 from Checkov, 14,033 from KICS, and 10,753 from Trivy.

B. Descriptive statistics

Vulnerability density is heavily right-skewed and declines as artifacts grow, making unmatched model–human comparisons misleading. After size matching, generated artifacts remain substantially denser than human templates, especially on the simplest tasks.

  • B. Descriptive statistics: Density distributions are heavily right-skewed and zero-inflated, motivating rank-based or negative-binomial inference.Fig. 1 displays the full distributions by configuration.
  • B. Descriptive statistics: 4.51 findings per resource in one-resource human files falls to 1.10 in files declaring twenty or more.Across generated artifacts, density has Spearman ρ = −0.55 with p = 1.7×10−78.
  • B. Descriptive statistics: 4.9× is the generated-to-human density ratio for one declared resource, decreasing to 1.4× for twenty or more resources.The largest-stratum comparison is not significant, with p = 0.058.
  • B. Descriptive statistics: 3.21×–3.87× is the aggregated density range above the human baseline across every model configuration.The band spans four vendors, open and closed weights, and three reasoning modes.
  • B. Descriptive statistics: The gap is largest on the simplest tasks, where minimal human templates are compared with substantially more flagged model output.The paper connects this pattern to structural over-generation.
  • B. Descriptive statistics: χ2 = 69.3 for simple strata and χ2 = 81.2 for complex strata lead Skillings–Mack to reject equal model performance.The corresponding p-values are 6.0 × 10−11 and 8.7 × 10−13.
  • B. Descriptive statistics: Complete-case Friedman testing retains zero usable blocks in both strata because realistic coverage gaps prevent complete configuration blocks.The paper therefore adopts Skillings–Mack for this benchmark design.

F. Post-hoc pairwise comparisons

Post-hoc comparisons reveal that cross-vendor differences exceed within-model reasoning-mode differences, while complexity interactions vary by configuration and require exposure-normalized interpretation.

  • 10 of 66 simple-stratum pairs and 9 of 66 complex-stratum pairs survive Holm correction.Pairwise tests use Wilcoxon signed-rank comparisons with pairwise deletion after the omnibus analysis.
  • In the simple stratum, gpt-4o and gpt-5-thinking occupy opposite ends and separate from the field’s middle.
  • In the complex stratum, Gemini configurations differ significantly from every Claude configuration, with claude-opus-4-6-thinking versus gemini-3.1-pro strongest at padj = 6.5×10−4.
  • Cross-vendor differences are larger than comparisons among reasoning arms of the same base model.No within-model reasoning-arm comparison survives correction.
  • Five configurations show significant complexity interactions, with gpt-5-thinking strongest at IRR 0.55 and Anthropic extended thinking stable at IRR 0.99.All significant interactions have IRR below 1; the gpt-5-thinking result has p = 4 × 10−4, while Anthropic extended thinking has p = 0.955.
  • The corrected gpt-4o interaction is not significant, overturning an earlier 14× simple-task vulnerability estimate produced without an exposure offset.The corrected interaction has p = 0.275.

H. Reasoning modes

Vendor extended thinking reduces vulnerability density relative to standard generation and prompted chain-of-thought, whereas prompted chain-of-thought alone shows no significant improvement. The effect is bounded because extended thinking is barely used during IaC generation.

  • −12.0%, vendor extended thinking significantly outperforms prompted chain-of-thought on the simple stratum.Extended thinking also reduces density versus standard generation by −13.2% (p = 0.012).
  • −1.3%, prompted chain-of-thought is statistically indistinguishable from standard generation.The comparison has p = 0.238.
  • The complex-stratum contrasts point in the same direction but are underpowered, with roughly half the paired observations and no significant results.The observed extended-thinking effects are −14.2% and −16.9%; larger complex-stratum samples are needed to distinguish these from null results.
  • Under 1% of the output budget is spent on reasoning tokens for complex tasks, limiting the achievable effect.Median reasoning expenditure is 29 tokens on simple scenarios and 151 on complex ones, versus completion medians of 886 and 18,533 tokens.
  • The rate model finds IRRs from 0.73 to 4.64 against a claude-opus-4-6 reference, with phi3 highest at IRR 4.64.The prompted-CoT arm is below the reference at IRR 0.73 (p = 0.041), while phi3 has p = 2.9 × 10−14.

K. Schema validity

Schema validity is low for frontier configurations and collapses for small local models, while severity counts and density require careful interpretation because parser-dependent denominators and output volume confound comparisons.

  • Schema validity: 27%–35% of frontier configurations pass schema validation, versus 8% for mistral, 6% for llama3, and 5% for phi3.Complex scenarios require multi-service architectures satisfying real provider schemas, making the absolute rates low.
  • Severity: All 14,017 Checkov findings have UNKNOWN severity and are excluded from the severity-tier analysis.Trivy and KICS provide CVSS-style tiers, enabling analysis of the remaining findings.
  • Severity: 573 CRITICAL+HIGH findings place claude-opus-4-6 highest, followed by its CoT arm at 503 and extended-thinking arm at 473.Absolute counts are confounded by output volume; per-resource analysis remains the basis for cross-configuration claims.
  • IaC format: Cross-format vulnerability densities are not comparable because resource-count denominators differ sharply by parser and IaC format.Mean parsed resource counts are 23.17 for Terraform, 2.31 for ARM, 1.15 for Kubernetes, and 0.90 for CloudFormation.
  • Structural context: Generated resource-count distributions sit to the right of the human corpus, indicating systematic structural over-generation.The supplied figure caption specifies the distributional direction; format-normalized counting remains future work.

N. Structural divergence

Generated IaC differs structurally from human-authored IaC across every tested metric, while human security evaluation shows limited agreement and automated judges are reliable mainly for factual verification.

  • Structural divergence: 36/36 KS tests reject equality between generated and human structural metrics at p < 0.05.Generated templates average 13–36 resources, compared with 5.31 for humans.
  • Structural divergence: Models systematically over-generate infrastructure relative to human engineers, but over-generation alone does not explain their higher per-resource security gap.Density normalizes by resource count, so the finding separates code volume from findings per unit of infrastructure.
  • Human evaluation: Human raters show negligible agreement on security-test relevance, with Fleiss’ κ = 0.059.Agreement is fair for plausibility and hallucination flagging, but only slight for architectural coherence.
  • Judge calibration: The LLM judge reaches 94.4% exact hallucination-detection agreement but only 27.8% exact architectural-coherence agreement.The reported boundary is factual verification rather than architectural assessment.

P. Two negative results

Two negative results constrain interpretation: deployability does not predict vulnerability, and complete-case Friedman testing cannot be computed for this incomplete benchmark design.

  • Deployability and security: Schema-validity pass rate and vulnerability density are uncorrelated: r = 0.158, p = 0.625.The same conclusion holds for Spearman ρ = 0.098, p = 0.761; only the low-parseability survivorship pattern for phi3 remains.
  • Incomplete-block inference: Complete-case Friedman testing retains zero blocks because no scenario contains all 12 configurations.The failure is uncomputability, not merely reduced statistical power.
  • Benchmark design: The simple/complex split is a design choice requiring validation, although KS tests show the tiers elicit materially different architectures.The tests reject equality for structural metrics across configurations.
  • Incomplete-block inference: Skillings–Mack is used for comparisons within complexity strata because realistic coverage gaps create incomplete blocks.The paper includes null and non-computable procedures to avoid selective reporting bias.

B. The gap is largest where review is weakest

The relative security penalty is greatest for small generated artifacts, which are less likely to receive careful review; reasoning modes improve results modestly but do not approach the human baseline.

  • Review and artifact size: 4.9× is the model-to-human vulnerability-density ratio for single-resource templates, falling to a non-significant 1.4× at twenty or more resources.The ratio declines monotonically with artifact size, reversing the intuition that complex generations are riskiest.
  • Review and artifact size: Review effort scales with artifact size, while relative security risk scales inversely, leaving small snippets with the largest relative penalty.Large generated architectures attract more scrutiny and are closer to human-authored code per resource.
  • Reasoning modes: Extended thinking reduces vulnerability by −13.2% and outperforms prompted chain-of-thought by −12.0%, p = 0.0013.No configuration moves near the human baseline despite the significant improvement.
  • Reasoning modes: Extended thinking consumes under 1% of the output budget on this task class, bounding the observed effect.The paper offers two consistent interpretations: IaC may elicit little reasoning, or reasoning training may not transfer to declarative configuration.
  • Practical implication: Enabling vendor reasoning is measurably better than prompting the model to think step by step, while prompted chain-of-thought provides no measurable security benefit.This guidance is narrower than treating reasoning as a solution to the human-model security gap.
  • Metric choice: Exposure-normalized rates with explicit denominators are recommended because absolute counts and omitted offsets confound security with code volume.The negative binomial model uses log(resources) as an exposure offset, while 130× overdispersion rules out Poisson.

E. Implications for evaluation methodology

The paper identifies evaluation practices that can distort conclusions about IaC security, including incomplete benchmark coverage, imperfect validation, scanner-scope limits, and measurement defects. It therefore emphasizes calibrated, transparent procedures and stronger controls for future studies.

  • Incomplete benchmark designs: Skillings–Mack should replace complete-case Friedman testing when realistic multi-model benchmarks contain missing scenario results.With twelve configurations, no scenario had complete coverage; refusals, quota exhaustion, and truncation create incomplete blocks.
  • Evaluation reliability: κ = 0.640 for factual verification versus κ = 0.177 for architectural judgment shows that LLM judges require per-criterion calibration against human raters.Human agreement on security-test relevance was near chance (κ = 0.059), further limiting assumptions about evaluation reliability.
  • Baseline construction: The human corpus is not a matched control because its templates target different scenarios and include curated examples that may be deliberately minimal.The resulting baseline bias could operate in either direction, and the paper does not claim to know which.
  • Static-analysis scope: Scanner findings indicate policy deviations rather than proven exploitability, so zero findings means only that the three rulesets flagged nothing.Severity analysis is limited because Checkov does not emit severity tiers without a commercial subscription.
  • Measurement infrastructure: Measurement defects produced plausible but incorrect results without errors, including aborted scans, discarded reports, schema-induced zero findings, and degraded resource counts.These failures motivate explicit validation of encoding, filenames, schemas, and resource-count sources.
  • External validity: Hosted-model changes, Terraform-heavy coverage, format-dependent engine rules, limited paired power (n ≤60), and only two reasoning-enabled model families constrain generalization.The reasoning contrasts therefore have a narrower evidentiary scope than the broader benchmark comparison.
Loading 2608.28021v1…