Last Secure Code Benchmark: Is the Code You Vibe Secure?

1 INTRODUCTION

• Users often check functionality but rarely inspect code, leaving implicit choices about untrusted input, sensitive data, and access boundaries unexamined.A functional document importer, for example, can still let a crafted archive write outside its directory.Source: §1 · p. 1
• Prior benchmarks each fall short on at least one of generated-code scope, language coverage, or vulnerability-category coverage.The paper positions LSCBench as addressing these dimensions together.Source: §1 · pp. 1–2
• Every task specifies functionality without naming the vulnerability or its mitigation, and separate functional and security suites grade implementations.This setup asks agents to handle security risks without explicit security guidance.Source: §1 · pp. 1–2
• LSCBench covers 450 tasks from 150 real vulnerabilities across seven languages, 24 CWE categories, and function, file, and repository scopes.Its CWE coverage includes 21 of the 2025 CWE Top 25 and 8 of the 2025 CWE Top 10 KEV Weaknesses.Source: §1 · pp. 1–2
• 73.6% to 91.3% of tasks pass functionally, but the best model passes both functional and security tests on only 25.1%.The best model passes both suites at all three granularities on only 12% of vulnerability scenarios.Source: §1 · p. 2

2 RELATED WORK

• Most code-generation benchmarks assess functionality rather than security, despite early studies finding about 40% of security-relevant completions vulnerable.Examples range from function-level suites to repository-level tasks.Source: §2 · p. 2
• Security benchmarks span single-snippet weakness prompts, repository-level tasks, and training methods including fine-tuning and security-reward reinforcement learning.Repository benchmarks include exploit-graded applications, CVE-project context, and vulnerability-fixing tasks with functional and security tests.Source: §2 · p. 2
• Figure 1 depicts a three-stage construction pipeline: task construction, patch-blind specification drafting, and fix-visible review with expert checks and task gates.The pipeline yields 150 scenarios and 450 tasks.Source: Figure 1 · p. 3
• LSCBench runs security tests against programs, and differs from prior benchmarks by covering realistic projects in seven languages, more crucial CWEs, and every vibe-coding setting.The paper notes that manual review does not scale, LLM judges can be fooled, and static analyzers match patterns rather than behavior.Source: §2 · p. 3

3 LAST SECURE CODE BENCHMARK

• Each scenario starts from a real project, known vulnerability, reference fix, functional tests, and a security test that replays a proof-of-concept attack.Function tasks fill masked vulnerable regions, File tasks implement affected files, and Repository tasks rebuild the declared source tree.Source: §3 · p. 3

3.1 BENCHMARK CONSTRUCTION

• CVE-Factory scenarios are selected from Top 25 weaknesses and cuttable at all three granularities; experts also hand-crafted 100 Java scenarios because Java is under 5% of CVE-Factory.The CVE-Factory candidates are ranked using fix-touched files, changed lines, and repository size; experts rewrote tests to assess behavior through proof-of-concept attacks.Source: §3.1 · p. 4
• Function tasks mask the smallest complete functions containing changed ranges, File tasks remove patched files, and Repository tasks empty a source scope capped at 8,000 lines removed.The remaining repository provides callers, build systems, and integration constraints; task environments are swept of answer-revealing artifacts and checked with reference and narrated-candidate controls.Source: §3.1 · p. 4
• A drafting agent writes each specification without the reference patch, from repository-level tasks down to files and masked regions, grounding statements in code without security guidance.The team sampled 30 scenarios to check consistency across granularities.Source: §3.1 · p. 4
• A fix-visible review agent and five experts revise specifications to require functionality without leaking or favoring the fix or vulnerable implementation.At the final gate, 247 of 254 scenarios pass oracle and nop controls, and experts select 150 scenarios; all 450 tasks pass both controls.Source: §3.1 · pp. 4–5

3.2 BENCHMARK OVERVIEW

• The final benchmark has 150 scenarios—115 public CVEs and 35 expert-crafted Java cases—covering seven languages and 24 CWE categories in 450 tasks.Coverage includes 21 of the 2025 CWE Top 25 and 8 of the 2025 CWE Top 10 KEV Weaknesses.Source: §3.2 · p. 5, Figure 2 · p. 5
• Function tasks retain target files and provide detailed inline contracts, File tasks remove target files with per-file specifications, and Repository tasks empty a declared source scope with a coarser repository specification.The task container includes the vulnerable project and build/run tools, while hidden functional and security suites run only in the verifier; Figure 3 summarizes visible inputs and verification.Source: §3.2 · pp. 5–6, Figure 3 · p. 5

4 EXPERIMENT SETUP

• Six frontier models ran with their default framework prompts and tools, interactive tools disabled, no network access, and a 500-turn cap.Five used Claude Code; GPT-5.6-Sol used Codex.Source: §4 · p. 6
• Tasks ran in a separate verifier that rebuilt the project, started its services, and executed both test suites after receiving only the agent's editable scope.Source: §4 · p. 6
• Figure 4 shows the share of 450 tasks passed and the share of 150 scenarios passed at all three granularities, separating functional-and-secure from functional-but-vulnerable outcomes.SECPASS and conditional rates are reported in Table 10.Source: Figure 4 · p. 7
• All metrics are pass@1: each task is attempted once, and FUNC∧SEC is the headline rate because security alone can reward implementations that do nothing.Scenario-level rates count success only when the model passes at all three granularities.Source: §4 · p. 7

5 RESULTS AND FINDINGS

• Five of six models pass the functional suite on about 90% of tasks, but Claude Fable 5 is both functional and secure on only 25%.The gap represents implementations that work while retaining the vulnerability.Source: §5 · p. 7
• The best model secures all three tasks in only about 12% of scenarios, although most models complete all three tasks functionally in more than half of scenarios.Source: §5 · p. 7
• 25.1% of tasks are secured by the best single model, versus 44.0% secured by at least one of six; 252 tasks are secured by none.Only 19 tasks are secured by all six, while 62 are secured by exactly one.Source: §5 · p. 7

5.1 RESULTS BY GRANULARITY

• Functional completion falls as scope grows, while FUNC∧SEC is flat or rises from Function to Repo for every model; File is a local minimum for three models.GPT-5.6-Sol's Repo solutions are secure nearly a third of the time; a possible explanation is that module-level work lets agents choose data flow and checks, unlike filling an existing vulnerable structure.Source: §5.1 · p. 8, Figure 5 · p. 8

5.2 TURNS VERSUS FUNC∧SEC

• The model using the fewest turns, Claude Fable 5, secures the most tasks; within models, secure implementations usually take about as many turns as functional-but-vulnerable ones.Qwen3.8-Max and GLM-5.3 spend markedly more turns on secure Repo solutions.Source: §5.2 · p. 8, Figure 6 · p. 8

5.3 WEAKNESS TYPES AND LANGUAGES

• Memory-safety defenses are secured most often, while policies absent from the specification—such as which data to withhold or destinations to refuse—are secured least often.The authors interpret the pattern as agents including local defenses while rarely reasoning about how code will be used and attacked after deployment.Source: §5.3 · pp. 8–9
• C, Python, and JavaScript code is secured on 27% to 30% of tasks, compared with 10% to 18% for Java, Go, and PHP; scenario weakness mixes explain much of the difference.C scenarios are almost all memory safety, while Java, Go, and PHP scenarios are dominated by injection, access control, and file handling.Source: §5.3 · p. 9

5.4 FAILURE PATTERNS

• Audits of 120 failed trajectories found distinct model patterns, including contract-literal behavior, vulnerability replay, weak defensive basics, default-allow shortcuts, and execution failures.For example, Kimi-K3 may dismiss a security anomaly surfaced by its own tests, while Qwen3.8-Max's failures are dominated by deadlocks, regex backtracking, and quadratic rendering paths.Source: §5.4 · pp. 9–10

6 CONCLUSION

• LSCBench evaluates 450 tasks derived from 150 real vulnerabilities across seven languages, 24 CWE categories, and three code-generation granularities.Source: §6 · p. 10
• Even the best model produces code that is both functional and secure on only 25% of tasks.Source: §6 · p. 10
• The authors say the results raise concern about deploying agent-written code without careful review.Source: §6 · p. 10

A DATASET COMPOSITION

• The composition tables describe language and CWE distributions, while scenario counts and task counts are distinct: each scenario contributes three granularity tasks.The reported 24 CWE categories should not be read as 24 entries from the Top-25 list.Source: A DATASET COMPOSITION · p. 14, Table 4 · p. 14, Table 5 · p. 14

B CONSTRUCTION DETAILS

• Construction work—including test authoring, specification drafting, and review—was run under Claude Code with Qwen3.8-Max (0902).Source: B CONSTRUCTION DETAILS · p. 15

B.1 SCENARIO SELECTION

• Scenario selection drew eligible CVEs by language and CWE coverage, combining an uncapped hardest-first draw with a CWE-capped draw to yield 226 CVEs.Language balance precedes CWE balance because weakness classes are tied to languages; 174 CVEs occur in both draws.Source: §B.1 · p. 15

B.2 TEST AUTHORING

• Autonomous agents authored behavioral functional and security suites that exercised each program through a shared external surface; 215 of 226 CVEs passed four-state validation.The process selected 326 scenarios, and 315 with validated tests entered Stage 1.Source: §B.2 · p. 16

B.3 BUILD RECIPES AND VERDICT REPRODUCTION

• Each verifier rebuilds the project from frozen scripts and checks that vulnerable code scores functional=1, secure=0, while reference-patched code scores 1,1.This distinguishes targeted security behavior from build or initialization faults.Source: §B.3 · p. 16

B.4 CUTTING A SCENARIO AT THREE GRANULARITIES

• Function tasks mask complete callables containing changed lines, File tasks remove patched files, and Repo tasks empty a declared source scope while retaining build and infrastructure substrate.Repo scope is chosen widest-first under caps of 8,000 source lines and 50 files; its layouts range from whole repositories to patched files alone.Source: §B.4 · p. 16

B.5 ENVIRONMENT SWEEP

• Before delivery, the environment sweep removes compiled outputs, installed copies, package caches, and Git history while preserving dependency trees; a broken build disqualifies the scenario.Source: §B.5 · p. 17

B.6 PER-GRANULARITY CONTROLS

• Each granularity must pass two controls: the restored reference solution must score 1,1, and a vulnerable candidate with the fix pasted as comments must fail security testing.Only scenarios retaining all three granularities were eligible for selection; 174 CVE and 93 Java scenarios left Stage 1 with complete triples.Source: §B.6 · p. 17

B.7 SPECIFICATION DRAFTING

• Specifications are drafted repository-first, then file-level, then region-level, with each narrower artifact grounded in the wider context.The writer first summarizes how tests and retained callers reach the reconstructed code, without stating expected outcomes or giving security guidance.Source: §B.7 · p. 17

B.8 REVIEW, GATE, AND SELECTION

• A second agent and five experts review specifications for vulnerable-design requirements, fix leakage, missing interfaces, and copied test inputs or outcomes.Thirty tasks received independent second-expert review, with more than 93% agreement; disagreements were settled by discussion.Source: §B.8 · p. 18
• Review and granularity controls leave 166 CVE and 88 Java scenarios reaching the final gate with all three granularities.Source: §B.8 · p. 18
• All 450 selected tasks pass both controls, after experts choose 150 scenarios from 247 that pass the gate and balance languages and vulnerability coverage.The gate runs 762 tasks with an oracle that must pass both suites and an empty agent that must pass neither; seven scenarios are withdrawn after empty-agent exceptions.Source: §B.8 · p. 18
• Table 7 lists the weakness classes prioritized during final selection in priority order, with C and C++ sharing one list.Source: Table 7 · p. 18

• Table 8 compares security oracles by describing what each evaluation observes and how it judges whether code meets a targeted security property.Source: C SECURITY EVALUATION IN RELATED… · p. 19
• Unit tests can run proof-of-concept attacks, while string assertions can check protocol or leakage properties.Source: C SECURITY EVALUATION IN RELATED… · p. 19
• The table's examples combine reported methods with selected scoring-code observations and do not establish an overall benchmark-quality ranking.Source: Table 8 · p. 19

• Table 9 lists the framework and reasoning effort used for each model, with effort set to each model's highest exposed level.Source: D EVALUATION SETTINGS · p. 19, Table 9 · p. 19
• Five of the six models run at max reasoning effort.Source: D EVALUATION SETTINGS · p. 19
• Qwen3.8-Max runs at xhigh because its scale has no higher level.Source: D EVALUATION SETTINGS · p. 19

E ADDITIONAL RESULTS

• Table 10 reports S | F among each model's functional successes and Common-F on the 271 tasks all six models pass functionally.Source: Table 10 · p. 20
• Table 11 groups outcomes by eight CWE weakness families and pools results across the six models, with italic rows showing family totals.Source: Table 11 · p. 20
• On the 52 scenarios all six models solve functionally at every granularity, GPT-5.6-Sol's FUNC∧SEC rises from 23.1% at Function to 42.3% at Repo, while DeepSeek-V4-Pro stays near 25%.The fixed-set rates make the comparison comparable across models and granularities; model-specific functional sets otherwise range from 75 to 124 scenarios.Source: E ADDITIONAL RESULTS · p. 20, Table 13 · p. 21
• Table 12 reports FUNCPASS and FUNC∧SEC by granularity across all 150 scenarios, supplying the values behind Figure 5.Source: Table 12 · p. 21
• The 40 (F, S) = (0, 1) outcomes receive no joint credit, and seven missing scores remain separate from 306 observed (0, 0) outcomes.The labels alone cannot establish whether security holds in an incomplete application or whether an application failure escapes the security tests.Source: Table 14 · p. 21, E ADDITIONAL RESULTS · p. 22
• Across all tasks, 252 are secured by no model, 62 by exactly one, and 19 by all six; this union indicates complementary coverage, not an advance selection policy.Table 15 gives overlap counts by granularity and across all 450 tasks.Source: E ADDITIONAL RESULTS · p. 22, Table 15 · p. 22
• Every model leaves most scenarios unsecured at all three granularities, while Table 16 counts scenarios secured at zero, one, two, or all three.Source: E ADDITIONAL RESULTS · p. 22, Table 16 · p. 22

F.1 A SOLVED TASK: CLAUDE FABLE 5 REBUILDS A MOUNT SERVICE

• 12 of 12 functional and 3 of 3 security tests pass when Claude uses ProcessBuilder arguments rather than a shell string.Qwen and Kimi pass the functional suite but leave injection open by building a shell command.Source: Figure 9 · p. 23, §F.1 · p. 24

F.2 A FUNCTIONAL BUT VULNERABLE TASK: KIMI-K3 REBUILDS A CONFIGURATION IMPORTER

• All 14 functional tests pass, but all four hostile YAML gadget tests execute under Kimi’s default YAML constructor.Kimi’s 33 self-checks probed well-formed input, not hostile archives; Claude and GPT-5.6-Sol use SafeConstructor and pass the security tests.Source: §F.2 · p. 24

F.3 A FAILED TASK: DEEPSEEK-V4-PRO DECLARES VICTORY WITHOUT A BUILD

• A nonexistent nextEntry() call prevents compilation and leaves both suites at zero; Kimi and Claude instead compile offline from the same cache.DeepSeek-V4-Pro stopped after the network-blocked Maven build and reread its file; had it compiled, its default YAML constructor would have left the gadget vulnerability open.Source: §F.3 · pp. 24, 26, Figure 10 · p. 25

F.4 A MEMORY-SAFETY TASK: QWEN3.8-MAX RESOLVES A DERIVED IMAGE IN LIBHEIF

• Five of six models pass both suites on this CWE-125 task, with Qwen’s loop stopping when the reference list is empty.GPT-5.6-Sol omits that check and crashes on malformed inputs; GLM-5.3 also bounds the loop to terminate cyclic references.Source: Figure 11 · p. 26, §F.4 · pp. 26–27

F.5 A SERVER-SIDE REQUEST FORGERY TASK: GPT-5.6-SOL FETCHES ANY URL IT IS HANDED

• All 12 functional tests pass while all 11 security tests fail because resolved file paths are not confined to the models directory.The reference fix checks containment after symlink resolution; the agent’s own fixtures stayed inside the directory.Source: Figure 12 · p. 27, §F.5 · pp. 27–28

F.6 AN INFORMATION-EXPOSURE TASK: GLM-5.3 PRINTS THE PASSWORD WITH THE REST OF THE CONFIGURATION

• All 11 functional tests pass, but five of six security tests fail because sibling-style tags expose passwords in configuration and SBOM output.The reference fix uses yaml:"-" and json:"-" so the loader still reads the fields while renderers omit them.Source: Figure 13 · p. 28, §F.6 · pp. 28–29