Source-linked AI summary
Evaluating and Preventing Security Smells in AI-Generated Ansible Code
Pandu Ranga Reddy Konala, Vimal Kumar, David Bainbridge, Junaid Haseeb
TL;DR
AI-generated infrastructure code can contain security smells that propagate into deployed systems, but evidence on its compliance with security requirements is limited. This paper evaluates 16 models across 278 Ansible roles and finds that structured prompting enables 4 models to generate compliant code, with the leading model achieving 95.2%–100% CIS compliance.
Problem
The study addresses limited evidence on whether AI-generated Infrastructure as Code meets security requirements, despite security smells propagating into insecure deployed systems.
Method
The authors evaluate 16 AI models generating Ansible for Tomcat and MongoDB, comparing 278 roles with 119 human-written roles using security-smell analysis and CIS benchmarks.
Results
4 of 16 models generated syntactically correct, quality-compliant code, and the leading model achieved 95.2%–100% CIS compliance versus 23%–43% for human-written code.
Takeaways & Limitations
For capable models, structured prompting can embed quality and compliance requirements into single-generation code production without iterative post-generation remediation.
Takeaways & Limitations
The evaluation covers two technologies, CIS Level 1 controls, Ansible, and zero-shot interactions, so results may differ for newer models or broader settings.
Abstract
from arXiv · showhide
AI coding assistants generate Infrastructure as Code, yet no work has examined whether this code meets security requirements. This matters because security smells in infrastructure code propagate to deployed systems, producing infrastructure that is insecure and untrustworthy. We evaluate 16 AI models generating Ansible roles for Apache Tomcat v10 and MongoDB v7, analysing 278 Ansible roles against CIS benchmarks. Without security guidance, all 16 AI models produced code containing security smells, resulting in vulnerable infrastructure that fails compliance verification and underperforms code written by human developers. We introduce an approach integrating Ansible best practices and CIS benchmarks into prompts through an extended CO-STAR framework, enabling security smell prevention during synthesis rather than detection after deployment. When this approach is applied, 4 out of 16 models generate compliant code, with the leading model achieving 95%-100% CIS compliance, a fourfold improvement over humans at 23%-43%, with overall code quality improving by 19%-49%. The remaining 12 models fail not because they cannot generate code but because they cannot follow instructions with multiple constraints. For capable models, the approach requires no retraining and can be adopted through system prompts.
1 Introduction
Infrastructure as Code automates deployment but can propagate security smells that undermine compliance, while AI-generated IaC compounds these risks. This study evaluates AI-generated Ansible code and embeds Ansible best practices and CIS benchmarks into generation prompts to prevent security smells during synthesis.
- Motivation: IaC replaces manual provisioning with version-controlled specifications, but security smells such as hardcoded credentials and excessive permissions can make deployed infrastructure insecure.Configuration-management tools including Ansible, Terraform, and Puppet translate IaC specifications into deployed infrastructure.
- Motivation: Code containing security smells fails compliance verification and cannot receive deployment authorisation under frameworks such as CIS Benchmarks and DISA STIGs.
- Motivation: 45%–62% of AI-generated code contains vulnerabilities, while IaC misconfigurations account for 68% of cloud security incidents.Infrastructure misconfigurations can become immediate attack vectors and may pose greater security threats than application vulnerabilities.
- Problem: Existing IaC security approaches detect smells after code enters repositories, leaving AI-generated code dependent on costly refinement cycles and security expertise for remediation.Detection cannot prevent security smells from entering codebases.
- Study scope: The study evaluates 16 foundational AI models generating Ansible code for Apache Tomcat and MongoDB deployments, assessing 278 roles against CIS benchmarks.The research asks whether AI-generated infrastructure has a viable default security posture and whether compliance requirements can be embedded directly into generation.
- Contribution: The proposed approach extends CO-STAR prompts with Ansible best practices and CIS benchmarks as generation constraints, preventing security smells during synthesis without model retraining for capable models.
2 Background
Compliance frameworks define security objectives that organisations must translate into technology-specific configurations, while automated tools verify compliance after deployment. AI-generated IaC may reproduce insecure training patterns because models are not reliably guided to distinguish secure configurations, a gap existing work has not empirically examined for compliance.
- Compliance foundations: Compliance frameworks specify security objectives without prescribing implementations, requiring organisations to translate them into configurations such as permissions, authentication, and encryption.Technology-specific implementation guides bridge the gap between abstract objectives and concrete security settings.
- Compliance foundations: CIS benchmarks organise controls into three Implementation Groups, with IG1 defining minimum cyber hygiene and IG2–IG3 adding protections for sophisticated threats.Each control includes security requirements, audit procedures, and remediation steps.
- Compliance foundations: Automated scanners assess running systems after deployment, making security verification a production gate for IaC-generated deployments.CIS-CAT Pro assesses CIS controls, while SCAP Compliance Checker provides equivalent functionality for STIGs.
- AI-generated IaC: AI models trained on public repositories encounter code ranging from hardened implementations to snippets containing security smells, while training rewards syntax and functionality rather than security.Models consequently have limited mechanisms to distinguish secure configurations from IaC with security smells.
- Research gap: Existing research covers compliance management, AI code security, and IaC security analysis, but none empirically examines compliance in AI-generated infrastructure code.Prior IaC studies identify security smells after code enters repositories and assume human developers implement corrections.
3 Baseline Security Performance of AI Models
The baseline evaluated 16 AI models generating Ansible roles without security guidance against 119 human-written roles. All models produced security smells, and AI-generated code consistently underperformed human implementations despite generally stronger structural conformity.
- Evaluation Design: 16 AI models were evaluated with identical zero-shot prompts, without examples or security guidance, against 119 human-written Ansible roles.Models operated standalone, with web search and retrieval-augmented generation disabled.
- Baseline Results: 4.56 was the mean total quality score and 0.634 the mean security score; no model exceeded both means.Claude Opus 4 exceeded the quality mean at 5.07, while Perplexity Sonar exceeded the security mean at 0.669.
- Baseline Results: 7 models citing domain-specific standards averaged 4.08 quality and 0.599 security, while 6 citing no standards averaged 3.95 quality and 0.618 security.These categories reflected claimed standards awareness rather than verified knowledge.
- Security Smells: 0 models implemented Ansible Vault, only 1 implemented error handling, and 6 failed to configure file permissions.These omissions corresponded to hardcoded credentials (CWE-798), improper exceptional-condition handling (CWE-755), and permission weaknesses (CWE-276).
- Security Smells: 16 models produced code containing security smells, while 10 of 16 produced syntax-error-free code and all followed directory conventions.Structural correctness therefore coexisted with hardcoded credentials, missing error handling, and absent access controls.
4 Methodology
The methodology embeds Ansible best practices and CIS requirements into generation prompts, translating compliance into measurable code attributes and verification constraints. It evaluates the extended framework across 278 Tomcat and MongoDB roles using CIS Level 1 controls and isolated deployments.
- Compliance-guided generation: Compliance-guided generation embeds Ansible best practices and regulatory requirements during synthesis instead of detecting violations after code generation.The approach makes quality and compliance explicit and maps regulatory requirements to measurable code-security attributes.
- Experimental design: The study uses zero-shot generation to reduce variation from example selection and iterative refinement, enabling direct comparisons across AI models.The same 16 models from the baseline evaluation were used, and all roles were deployed in isolated test environments.
- Prompt framework: The extended CO-STAR framework adds QUALITY requirements for Ansible practices and COMPLIANCE requirements for CIS controls.QUALITY includes error handling and Ansible Vault credential management, while COMPLIANCE specifies service configurations and access restrictions.
- Prompt framework: Categories A, B, and C classify requirements as mandatory, context-dependent, and anti-patterns to avoid, while verification requires models to validate all requirements before completion.Anti-patterns include hardcoded passwords and insecure defaults.
- Experimental design: 278 Ansible roles span 135 Tomcat roles and 143 MongoDB roles, combining AI-generated and human-written implementations evaluated against CIS Level 1 controls.Tomcat roles were assessed against 21 controls, while MongoDB roles were assessed against 7 controls.
5 Results and Discussion
Only four of 16 models generated syntactically correct, quality-compliant, and CIS-compliant Ansible roles, while structured compliance-aware prompting substantially improved compliance and quality. Success depended on multi-constraint instruction following, with code quality correlating with compliance and exceeding human-written baselines.
- Model Success: 4 of 16 models generated syntactically correct code satisfying quality and compliance constraints, with three from the appropriate standards category and one from the no-standards category.None of the models from the inappropriate standards category succeeded.
- Instruction Following: 75% of models failed despite prompts of only 1,572–3,403 tokens, below 3% of the smallest 128K-token context window.Claude Sonnet 4 failed despite an 80.2% SWE-bench score, whereas Pixtral Large succeeded without popular benchmark data; Llama 4 failed with 10M tokens while Claude Opus 4 succeeded with 200K tokens.
- Mechanism and Limitation: The framework tests whether models can parse layered prompts, interpret A/B/C priorities and attribute-specific syntax, and apply constraints across quality and compliance domains.All four successful models were closed-source, limiting investigation of the factors underlying success without provider collaboration.
- Quality and Compliance: Higher pre-deployment structure/security scores corresponded to higher compliance rates, and MongoDB’s 7 controls enabled higher compliance than Tomcat’s 21 controls.The four successful models achieved competitive quality rankings, and manual inspection found none of the baseline security smells; all implemented Ansible Vault for credential management.
- Human Baseline: 113 of 119 Tomcat roles (95.0%) and 118 of 127 MongoDB roles (92.9%) lacked configuration templates, while implementing human roles reached maximum compliance of 23.8% and 42.85%, respectively.Human-role failures included incorrect permission syntax and malformed configuration.
- Compliance Outcomes: 95.2%–100% CIS compliance was achieved by the leading model through single-generation structured prompting, compared with 23%–43% for human-written code.The extended CO-STAR framework embeds quality and compliance requirements during generation, avoiding iterative post-generation remediation.
6 Conclusion
Evaluating 16 AI models across 278 Ansible roles found that all models produced security smells by default and underperformed human-written implementations. Structured prompting enabled four models to generate compliant, smell-free code, with the leading model achieving 95%–100% CIS compliance and overall quality improving by 19%–49%.
- Default generation: 16 AI models evaluated across 278 Ansible roles all produced security smells by default and underperformed human-written implementations.The evaluation examined whether AI models could generate quality-compliant infrastructure as code.
- Instruction following: 12 out of 16 models failed because of instruction-following limitations with multiple constraints, rather than coding inability.Models achieving 80% on SWE-bench failed, while others lacking benchmark data succeeded.
- Structured prompting: Four models generated compliant code with structured prompting, while the leading model achieved 95%–100% CIS compliance versus 23%–43% for human implementations.Overall code quality improved by 19%–49%.
- Compliance verification: Manual inspection confirmed that the four successful roles contained no previously detected security smells, providing a foundation for compliance.All successful models were closed-source.