When Talk Isn't Code: Comparing LLM Agents That Simulate and Develop Software

1 Introduction

• LLMs both simulate network services and write them, while prior work reports vulnerabilities in generated code and security failures missed by functional tests.The paper cites about 40% of 1,689 GitHub Copilot programs as vulnerable.Source: §1 · p. 1
• Prior studies assess security knowledge, simulated services, and generated programs separately; no study had compared a model’s conversation with the program that same model builds.In a preliminary comparison, the probe over-reported checks in 22 of 26 pairs where the program lacked them, and S-mode over-reported in 10.Source: §1 · p. 1
• The study uses RFC-documented network protocols whose message-based attacks can be replayed in conversation or against code and judged deterministically.The authors caution that these properties do not apply to every kind of software.Source: §1 · p. 2
• A harness gives both modes shared documents, attack cases, and an oracle, then evaluates 15 models across four protocols and multiple context levels.The probe serves as a baseline, and the richest S-mode and D-mode settings both hold the full security specification.Source: §1 · pp. 2–3
• With the full security specification, median attack success is at most 4% on reassembly, HTTP, and firewall implementations in both modes, but 13% in S-mode and 15% in D-mode on DNS.No model reaches 0% on DNS.Source: §1 · p. 3

2 Background: Four Protocols and What Their Attacks Require

• DNS resolution and IP reassembly are stateful: resolvers must validate response details before caching, while gateways must handle overlaps, undersized first fragments, and incomplete datagrams.The DNS checks include transaction ID, destination address, exact question, and randomized letter case; RFC 5722 requires dropping an entire datagram with overlapping fragments.Source: §2 · p. 3
• HTTP forwarding and firewall rule matching are stateless, making decisions from the current request or packet and a static rule list.Their attacks include request smuggling and firewall policy anomalies.Source: §2 · pp. 3–4
• The off-path adversary can send messages but cannot read internal state or forge cryptographic material; each vulnerability case targets a documented attack class.The cases include forged DNS answers, injected fragments, attacker-controlled HTTP headers, and crafted firewall packets.Source: §2 · p. 4
• DNS attacks that still succeed concentrate on 0x20 validation and the TTL cap, rather than transaction-ID checks.The suite also includes checks stored when a query is sent and checks readable from a single reply.Source: §2 · p. 4

3.1 Harness and Models

• A Python harness replays scripted cases against either a conversational model or a generated program, with programmatic verdicts and a network-isolated sandbox for D-mode.The same case scripts are also run against a reference implementation for functional differential testing.Source: §3.1 · p. 4, Figure 2 · p. 5
• The study evaluates 15 LLMs through the same Claude Code agent, keeping its system prompt, tools, and agent loop identical across models.Models are reached through Anthropic-compatible endpoints.Source: §3.1 · p. 5

3.2 Probe and Modes

• The probe asks an open question without specifications, tests, or cases; S-mode plays the implementation turn by turn, while D-mode writes a Python program.At P3, S-mode receives its full state on every turn; D-mode’s richer levels add shell access, public functional tests, and repair rounds, but those tests contain no attacks.Source: §3.2 · pp. 5–6, Table 1 · p. 6

3.3 Test Suite and Oracle

• The 51-case suite includes functional, vulnerability, and injection tests; each vulnerability case targets a specified security check, and trace-based checkers determine attack outcomes.The secure reference and single-check mutants validate that each vulnerability case targets its intended check; the caption notes that V-FW-004 needs no recall of earlier messages.Source: §3.3 · pp. 6–8, Table 2 · p. 7

3.4 Procedure

• Every model, protocol, and level cell runs three times in independent sessions, producing 9,360 S-mode case runs and 8,661 D-mode case runs over 641 programs.The study pools repetitions and reports Wilson 95% confidence intervals; runs with infrastructure failures or inconclusive verdicts are excluded from ASR.Source: §3.4 · p. 8

4 RQ1: Does Layered Prompting Reduce S-Mode Attack Success?

• The RQ1 matrix tests 15 models across four protocols and four S-mode context levels, with three fresh-session repetitions per setting.Each conversation and generated program faces all vulnerability cases for its protocol.Source: §4 · p. 8
• For DNS, ASR falls from 17–63% at P0 to 12–42% at P3 but never reaches zero.For three models, adding protocol documentation at P1 raises DNS ASR; MiniMax-2.5 rises from 63% to 82%.Source: §4 · p. 9
• On reassembly, median ASR rises from 37% at P0 to 52% at P1, then falls to 4% at P2; five models reach zero and stay there at P3.Only Kimi-2.6, MiniMax-2.5, and MiniMax-3 retain nonzero reassembly ASR at P3.Source: §4 · p. 9
• At P3, 13 of 15 models reach zero ASR on HTTP and all 15 do on firewall; the security specification, unlike functional documentation alone, brings both toward the floor.At P1, no model reaches zero on firewall and only six do on HTTP.Source: §4 · p. 9
• At P3, median S-mode ASR is 0–4% on reassembly, HTTP, and firewall, while DNS remains at 12–42% for every model.The reported pattern rules out statefulness alone as an explanation: reassembly is stateful but reaches the floor.Source: §4 · p. 9, Figure 3 · p. 9

5 RQ2: Does the Same Pattern Hold When Models Write Code?

• At D2, DNS ASR remains 11–23% for every model that produced code, while firewall reaches zero and 10 of 14 models reach zero on reassembly.HTTP ASR is zero for all but Kimi-2.6 and MiniMax-2.5, both at 25%.Source: §5 · p. 10, Table 3 · p. 10

5.1 Security of the Programs

• Kimi-2.6’s three D1 reassembly programs pass the public suite, yet 8/9, 8/9, and 6/7 conclusive attacks succeed; D2 attacks succeed 0, 2, and 0 times out of 9.D1 code sorts and concatenates fragments without checking overlap, while D2 code explicitly checks overlap and drops affected datagrams.Source: §5.1 · p. 10
• MiniMax-2.5’s HTTP ASR rises from 25% at D0 to 42% at D1, then returns to 25% at D2, despite passing all functional HTTP tests at D1.The reported repair loop optimizes for passing tests, not security, and can introduce a weakness on header-rewriting paths.Source: §5.1 · p. 10

5.2 Development Effort and Functional Correctness

• A strongest-setting evaluation costs a median of $3.00 and 7.7 minutes in S-mode versus $1.65 and 16.5 minutes for three D2 programs; S-mode is cheaper in dollars in only 21 of 57 cells.S-mode saves time and avoids the sandbox, but a generated program can be attacked by every case after one build.Source: §5.2 · pp. 10–11
• Every D1 and D2 program passes its public tests in the first harness round, yet D2’s median ASR is 0% on firewall, HTTP, and reassembly and 15% on DNS.Functional success does not establish security; by D2, every model’s DNS program passes at least 88% of functional cases.Source: §5.2 · pp. 10–11

6.1 Knowledge and the Residual ASR

• In 26 check-model pairs where D-mode programs lacked a check, models named it in 22, while S-mode held it in 10; for 42 safe pairs, the probe named all checks.The probe flags 4 of 26 vulnerable pairs, versus 16 for S-mode; the two residual DNS checks, 0x20 and TTL capping, account for 84% of successful attacks across D-mode levels.Source: §6.1 · pp. 12–13

6.2 Experiment 1: Adding a Missing Requirement

• Adding the 0x20 requirement reduced V-DNS-009 attack success from 100% (42/42) to 34% (13/38), while the control check stayed at 24% (10/41 before; 9/38 after).Regenerated programs compared the echoed name against its wire form before accepting it; no original D2 program did.Source: §6.2 · p. 13

6.3 Experiment 2: Making a Stated Rule Concrete

• A worked example did not improve the already-stated bailiwick rule: V-DNS-004 succeeded on 2 of 12 runs both before and after.The authors interpret this residual as a limit on implementing a correctly stated cross-record rule.Source: §6.3 · p. 14

6.4 Experiment 3: Deleting a Stated Rule

• Deleting either reassembly rule raised attacks beyond its own cases; pooled success rose from 2% to 6% after overlap-rule deletion and to 5% after first-fragment-rule deletion.The authors report the asymmetry between adding and deleting rules as an observation whose mechanism remains unexplained.Source: §6.4 · p. 14

6 RQ3: Why a Known Defense Is Missing

• The probe does not predict implemented checks: adding a missing requirement changed only its named check, while deleting a stated rule weakened other checks too.Source: §6.2 · p. 13, §6.4 · p. 14

7.1 S-Mode as a Screening Test

• Across 115 DNS model-case pairs, S-mode reaches sensitivity 0.824 and specificity 0.959, yet the modes disagree on seven individual pairs despite similar totals.S-mode misses three vulnerabilities and raises four false alarms; three of 17 vulnerable pairs go unflagged.Source: §7.1 · pp. 14–15

7.2 Under- and Over-Reporting

• S-mode under-reports stateful checks that D-mode handles in code, but over-reports the TTL cap: 14 of 15 models held it in S-mode while seven D-mode programs omitted it.The examples include first-response-wins behavior across turns and resolvers that copy TTL values into the cache unchanged.Source: §7.2 · pp. 15–16

7.3 What S-Mode Measures and Misses

• S-mode can reveal whether a model carries a rule across turns, but it cannot flag models that fail to produce a program in D-mode.Qwen-Max-3.8 timed out on all nine DNS and all nine reassembly attempts at the 1,800-second budget, while still returning S-mode replies.Source: §7.3 · p. 16
• On DNS, only 87.9% of S-mode and 85.7% of D-mode (model, level, V case) triples had the same verdict across all three repetitions.The corresponding repeatability was at least 94% elsewhere, so a single run could misjudge about one DNS outcome in eight.Source: §7.3 · p. 17

7.1 S-Mode as a Screening Test

• S-mode flags 82% of real DNS vulnerabilities and leaves 96% of safe cases unflagged, but it both over-reports and under-reports checks relative to D-mode.The authors conclude that S-mode can screen but not replace D-mode testing.Source: §7.1 · p. 15, §7.3 · p. 17

Threats to Validity

• The verdicts rely on one automated checker per property, though the authors hand-reviewed reported excerpts and fixed two checker bugs.The checkers were validated against a reference implementation.Source: §8 · p. 17
• Binary attack success is only a proxy for exploitability in one scripted scenario, and tested phrasing may miss equivalent variants.Cases came from documented attack classes rather than attack traffic or adaptive search.Source: §8 · p. 17
• Mode comparisons use only P3 and D2, the strongest settings for two realistic workflows, because other protocols already include security properties in D-mode notes.This differs from S-mode at P0 and P1.Source: §8 · p. 17
• Specification experiments have 12–84 conclusive runs per condition, limiting detection of small effects; deletion results are observations, not confirmed effects.The reported 0x20 effect survives Bonferroni correction, unlike the deletion results.Source: §8 · p. 17
• External validity is limited to four network protocols and 15 models chosen by availability, leaving software with other interfaces untested.Two protocols per statefulness class cannot establish what generally explains residual DNS attack success.Source: §8 · p. 17

Security of LLM-generated code

• Earlier code-security studies evaluate generated programs alone, while this work asks whether simulated behavior predicts vulnerabilities in the same model’s code.Prior findings include vulnerable Copilot completions and code that passes functional but fails security tests [35].Source: §9 · p. 18
• No prior work, to the authors’ knowledge, fixes one task, attack suite, and oracle across S-mode, D-mode, and the knowledge probe.The study also examines specification changes and includes an S-mode control on the same cases.Source: §9 · p. 18

LLMs that simulate systems and people

• Studies of simulated systems examine role play and service behavior, but do not compare a simulation with the same model building that system.Prior simulator work reports limited style-dimension matches [56] and separates simulated from real execution [12].Source: §9 · p. 18

Validity of LLM4SE benchmarks and LLMs for networks

• Benchmark-validity critiques motivate reporting case-level agreement, while networking studies use LLMs as engineering tools rather than protocol implementations.Examples include unreliable task labels [40], merged end-to-end resolution scores [13], and audits of SWE-bench Lite.Source: §9 · p. 18

Lessons learned

• Programs passed public tests yet remained exploitable, so adversarial tests belong in continuous integration.The authors recommend judging the program an agent builds, not only its answers or conversation.Source: §10 · p. 18
• S-mode is a first filter, but about one real DNS vulnerability in six escapes it, so D-mode should test the checks S-mode misses.The probe cannot approve a model.Source: §10 · p. 18

Recommendations for researchers

• Specifications should name required checks, and each run should be repeated because about one DNS outcome in eight differs across repetitions.The authors also recommend stating which requirements the specification names.Source: §10 · p. 18
• Researchers should validate the oracle against a secure reference implementation.The study’s reference implementation resists every case and fails each when its target check is disabled.Source: §10 · p. 18

Ethical Considerations

• Attacks target only artifacts built for the study, against programs in a network-isolated sandbox; the artifacts are never deployed.The attacks encode long-documented weaknesses.Source: §11 · p. 19
• The study involves no human subjects or private data.Source: §11 · p. 19
• An LLM assistant was used only to polish the paper’s language, not to produce its design, experiments, analysis, or claims.Source: §11 · p. 19

• Only testing the built code shows what an agent ships.Source: §12 · p. 19
• The knowledge probe does not predict the program.Source: §12 · p. 19
• S-mode is a useful filter, but does not replace testing the built code.Source: §12 · p. 19