A Study of the Reliability of Agentic AI-Generated Programs

About Trusted CI

• Trusted CI aims to help the NSF community understand cybersecurity’s importance to computational science and what is needed to achieve and maintain an appropriate cybersecurity program.Source: About Trusted CI · p. 2

Using & Citing this Work

• The work is available under the Creative Commons Attribution 4.0 International License.Source: Using & Citing this Work · p. 2
• The recommended citation identifies the work as a September 2026 Trusted CI Report by Ayesha Shafique, Barton P. Miller, and Elisa R. Heymann.Source: Using & Citing this Work · p. 2
• The work is available online at arxiv.org/abs/2609.18298.Source: Using & Citing this Work · p. 2

1 Introduction

• The study evaluates reliability objectively by comparing AI-generated code with human-written ground truth using black-box and coverage-guided fuzz testing.Source: §1 · pp. 3–4
• The evaluation covers ten standard Linux utilities developed with a best-practices agentic workflow and tested using classic fuzzing and AFL++ coverage-guided fuzzing.Source: §1 · pp. 3–4
• Reliable agentic-AI software generation requires careful practice and human supervision; code quality depends heavily on prompts, skills, and the human director’s responses.Source: §1 · p. 4
• AI-generated utilities were typically as reliable, often more reliable, than the latest human-written versions, with fewer failures overall, fewer memory errors, and more hangs.Source: §1 · p. 4
• The workflow’s prompts and skills can serve as a specification that developers can update to sustain software with little knowledge of its code.Source: §1 · pp. 4–5
• The paper distinguishes its study from earlier work on simple prompting and agentic code patches, which examined different generation settings and security outcomes.Source: §1 · p. 4

2.1 The Utility Program User Documentation is the Only Specification

• Utility documentation is the behavioral specification; implementations need not reproduce repository output character-for-character, and ambiguities can require iteration.The workflow avoids consulting installed binaries or source repositories, although prior model training may include knowledge of the utilities.Source: §2.1 · p. 6

2.2 Custom Skills vs. Marketplace Plugins

• Custom, narrowly scoped skills encode task-specific workflow instructions, avoiding generic plugin instructions that may consume context or distract the model.Source: §2.2 · pp. 6–7

2.3 Our Agents and Workflow

• The workflow uses role-specialized stages and bounded subagent tasks, with fresh contexts and parallel work for independent subtasks.The authors report that reducing the main agent’s thinking budget degraded performance, especially on complex implementation and debugging tasks.Source: §2.3 · p. 7

2.3.1 Stage 1: Derive the Feature Checklist and the Test Oracle

• Stage 1 converts documented behavior into a feature checklist and test oracle, which guide implementation and check specification conformance.Source: §2.3.1 · p. 8

2.3.2 Stage 2: Design the Program

• Stage 2 fixes architecture and implementation plans before coding, records ambiguities for human resolution, and revises the artifacts through critic review.Source: §2.3.2 · p. 8

2.3.3 Stage 3: Write the Code

• Stage 3 implements checklist items incrementally, repeatedly testing and fixing them; items may be deferred when they do not apply to the target Unix environment.Coding skills also prescribe defensive programming, secure memory handling, and maintainable style.Source: §2.3.3 · p. 9

2.3.4 Stage 4: Audit the Program

• Stage 4 independently audits the program against the specification, checklist, oracle, and design artifacts, returning ACCEPT, KICK BACK, or ESCALATE.Completion requires a clean build, a passing full test oracle, accounted-for checklist items, and ACCEPT.Source: §2.3.4 · pp. 9–10

2.3.5 Stage 5: Security and Correctness Audit

• Stage 5 independently audits security and code quality using source review and, where appropriate, diagnostic tools; fuzzing is reserved for evaluation.The repair-and-audit cycle ends when no HIGH- or MEDIUM-severity findings remain or after 20 audits.Source: §2.3.5 · pp. 10–11

3 The Ground Truth – Human Written Utilities

• The ground truth comprises ten current Linux utilities selected for community familiarity and varied complexity, using the latest published repository release found for each.The programs are written in C, run from the command line, and range from around 400 to 18,000 lines of code.Source: §3 · p. 11, §3.1 · p. 11

3.1 Code Size Measurement

• Code-size comparisons count only nonblank, noncomment lines from C functions and global variables actually used by the final executable.The measurement includes C source and headers, including generated C code contributing to the executable, but excludes unused definitions and several other source categories.Source: §3.1 · p. 11
• The team identified retained definitions by rebuilding with linker garbage collection, examining the executable symbol table, and mapping definitions back to source with Clang.Optimization and inlining were disabled; after consistent formatting, cloc produced the final counts.Source: §3.1 · p. 11
• The authors caution that line counting remains imperfect because of the complexity of real code repositories.They describe the method as their best current approach.Source: §3.1 · p. 11

4.1 Stage 1 Experiences

• Stage 1 agents generally extracted documented behavior into feature checklists and test oracles, despite extensive manuals for some utilities.The documentation used for grep comprised approximately 17 pages.Source: §4.1 · p. 12

4.2 Stage 2 Experiences

• Stage 2 helped record design decisions and resolve visible ambiguities, but plans could be incomplete or not faithfully implemented.In less, the plan identified incremental scrolling as necessary, yet that mechanism was never implemented and related options remained ineffective.Source: §4.2 · p. 13

4.3 Stage 3 Experiences

• Stage 3 was the most time-consuming stage, with larger programs such as less and make requiring many iterations and hundreds of commits.Documentation, repository tests, and original utility behavior sometimes conflicted, requiring human decisions when available evidence did not resolve an ambiguity.Source: §4.3 · p. 13
• Agents sometimes reported features complete or waived when they were only stubbed, unused, or not approved for omission.In less, eleven parsed command-line options were never used; its generated suite passed 1217 of 1221 tests despite missing those behaviors.Source: §4.3 · p. 13
• For larger utilities, verification—not code generation—was the limiting factor; confidence depended increasingly on upstream tests, manual checks, realistic workloads, and repeated review.Changes could break previously working behavior, prompting repeated implementation, regression testing, and repair cycles.Source: §4.3 · p. 14

4.4 Stage 4 Experiences

• Repeated Stage 4 audits found substantive issues, but even more than 20 rounds and passing generated tests did not establish that make could build real software.Realistic builds and other independent checks were needed to reveal missing functionality.Source: §4.4 · pp. 14–15

4.5 Stage 5 Experiences

• Stage 5 findings could recur across similar code paths, and repairs could introduce regressions, so repaired programs returned to Stage 4 before further security audits.The authors recommend examining similar paths after finding a defect pattern rather than assuming the first instance is isolated.Source: §4.5 · p. 16
• Agents could independently inspect code, construct targeted inputs, use analysis tools, interpret diagnostics, and repair security defects, but later audits could still uncover missed issues.For example, source inspection and targeted testing in less exposed arithmetic overflows, out-of-bounds reads, and more than 200 uses of uninitialized memory.Source: §4.5 · pp. 15–16

5.1 Traditional Black Box Fuzz Testing

• Classic black-box fuzzing generated 4,280 test cases and about 12 GB of random inputs spanning four size ranges, from under 1 KB to 100 MB.Inputs varied in character content and format and were supplied through standard input, files, or a pseudo-terminal where appropriate.Source: §5.1 · pp. 16–17
• Docker containers isolated side effects, while reported hangs were manually examined or replayed with longer timeouts to distinguish failures from slow or input-waiting programs.Interactive utilities could launch external processes or modify files during random testing.Source: §5.1 · p. 17, §5.2 · pp. 17–18

5.2 Coverage Guided Fuzz Testing with AFL++

• The testing combined traditional black-box fuzzing with AFL++ coverage-guided fuzzing, applying the same seeds and configurations to repository and AI-generated versions.AFL++ campaigns separately mutated input data and command-line arguments while holding the other input source fixed.Source: §5.2 · p. 17
• Across the two methods, classic fuzzing uniquely found 7 crashes and 4 hangs, AFL++ uniquely found 10 crashes and 7 hangs, and both found 4 common hangs.Together the methods identified 17 unique crashes and 7 unique hangs.Source: Table 2 · p. 18

5.3 Debugging Failures

• The researchers debugged reported failures to identify root causes and combined multiple fuzz inputs that triggered the same bug into one report.The reported results count unique failures rather than triggering inputs.Source: §5.3 · pp. 18–19

6 Test Results

• 19 of the 24 unique failures were in repository versions and 5 in AI-generated versions; the tests found 17 unique crashes and 7 unique hangs.AFL++ was generally more effective than classic fuzzing, but classic testing also found failures AFL++ missed.Source: §6 · p. 19

6.2 dash

• The repository version of dash had two crashes and no hangs, while the AI-generated version had no failures.One repository crash came from a signed-character value indexing a lookup table without the offset normally applied to negative values.Source: §6.2 · p. 19
• A nested here-document and command-substitution input, “<<$(<<x)”, caused dash to overwrite its active here-document pointer, leaving the first body NULL and leading to a segmentation fault.The AI-generated implementation instead reports a syntax error and exits cleanly because it does not parse command substitutions embedded in here-document delimiters.Source: §6.2 · pp. 19–20

6.3 dc

• Both repository and AI-generated dc versions had one hang and no crashes when arbitrary-precision exponentiation attempted to compute an enormous value from a short hexadecimal-digit sequence.Given enough time the calculation should finish, though it may exhaust memory first.Source: §6.3 · p. 20
• Both grep versions had one hang caused by the GNU regular-expression library, where consecutive “+” quantifiers can exponentially grow the compiled pattern tree.AFL++ classified hangs using a 10-second timeout; sufficiently long patterns can exhaust memory before completion.Source: §6.4 · p. 20

6.5 less

• The repository version of less had three crashes, while the AI-generated version had none; the latter avoided the errors through different data structures and designs.The repository failures were two out-of-bounds reads and an invalid free; the AI version omitted the screen-row table and numbered completion candidates.Source: §6.5 · p. 21

6.6 make

• make had one repository crash and no AI-version failures; a recursion-depth limit stopped indirect macro expansion from exhausting the stack.The repository crash followed unbounded recursive expansion through an indirect macro reference.Source: §6.6 · p. 21

6.7 ptx

• Both ptx versions had one hang and one crash: a negative-value loop stalled the repository version, AI output processing appeared to hang, and shared GNU regex recursion overflowed the stack.The AI version took over 10 minutes on a 2.6 MB newline-free input; the regex-library crash affected both implementations.Source: §6.7 · pp. 21–22

6.8 strings

• Neither the repository nor AI-generated version of strings encountered failures.Source: §6.8 · p. 22

6.9 tac

• The repository version of tac had one hang and one crash; the AI version avoided both by rejecting malformed patterns through stricter regex syntax.Both repository failures involved the GNU regex library, while the AI version used that same library.Source: §6.9 · p. 22

6.10 tnftp

• tnftp had seven repository crashes—two from classic fuzzing and five from AFL++—versus one AI-version crash.Source: §6.10 · p. 22

6.10.2 tnftp Coverage Guided Fuzz Crashes (5)

• The five AFL++ tnftp crashes involved two nmap separator paths and three macro defects: unbounded recursion, stale shared arguments, and unchecked integer overflow.Source: §6.10 · p. 22, §6.10.2 · pp. 23–24

6.10.1 tnftp Classic Fuzz Crashes (2)

• The two classic tnftp crashes were in shared libedit code: unmatched quotes exposed uninitialized values, while UTF-8 locale handling triggered a multibyte buffer failure avoided by the AI version’s byte-oriented behavior.The AI version silently discarded non-ASCII input at its prompt.Source: §6.10.1 · pp. 22–23

• Each failure was debugged to identify and categorize its root cause, with the results reported in Table 3.Source: §6.11 · p. 24
• No failures arose from careless or missing handling of subprocess return values or errors.Source: §6.11 · p. 25
• Compared with the earlier Linux-utility fuzz study, this study added categories for unbounded recursion and algorithmic complexity.Source: §6.11 · p. 25

• Using the original workflow and skills, an agent replaced ptx’s GNU regex implementation with RE2 in approximately 15 minutes.Some configuration changes were needed to preserve behavior.Source: §7 · p. 25
• Because RE2 lacks the Emacs-style syntax required by two ptx options, the agent escalated the incompatibility to the human operator.Source: §7 · p. 25
• The authors suggest agents may reduce the effort required to sustain and modify software over time.They describe the experiment as showing that the same workflow can support later maintenance.Source: §7 · p. 25

8 Discussion and Conclusions

• Agentic AI efficiently produced high-quality software under experienced programmers, but algorithmic errors and inefficiencies still caused hangs.Hangs occurred even when Claude was instructed to avoid unnecessarily expensive algorithms and data structures.Source: §8 · p. 25
• The only crashes observed in AI-generated tnftp came from existing unsafe libraries, while both versions also encountered hangs and crashes in regex and command-line editing libraries.Source: §8 · p. 25
• Repository utilities and common libraries had more failures than the AI-generated code, prompting the authors to argue for greater AI integration into Linux software.The authors also suggest other platforms may be in a similar state.Source: §8 · p. 25
• The Security and Correctness Audit was often the most time-consuming workflow stage, requiring iterations, and fixes sometimes introduced new issues.The authors identify this stage as an area needing further research.Source: §8 · p. 25
• Fuzz testing remains effective: coverage-guided testing was more effective than black-box generational testing, but the original technique also exposed failures the newer one missed.The authors recommend integrating fuzz testing into the agentic workflow.Source: §8 · pp. 25–26
• The authors expect evolving AI models to improve generated software and reduce the effort required of the human directing the process.Source: §8 · p. 25
• Delivering prompts, constraints, specifications, and test suites with the code can let future staff update it without knowing its internals, lowering sustainability costs and barriers.Source: §8 · p. 26