Source-linked AI summary
Secure and Functional Code Generation with Self-Evaluation
Md Afif Al Mamun, Gias Uddin
TL;DR
LLMs can generate code that passes functional tests while remaining insecure, and security interventions can sacrifice required behavior. SecGent retrieves relevant CWE knowledge and uses functional and security tests to guide generation and repair. Across five models and five languages, it achieves the highest joint functional–security success among the evaluated inference-time baselines.
Problem
Functionally correct code can remain vulnerable, while security-oriented interventions may reject valid inputs or remove required behavior, making joint functionality and security important to assess.
Source: §1 · pp. 1–2Method
SecGent retrieves task-relevant textual CWE records to guide generation and candidate-independent functional and security tests, then uses execution feedback for bounded repair.
Source: §1 · p. 2Results
Across five models and five languages, SecGent achieves the highest joint functional–security success among evaluated inference-time baselines; open-weight mean FS@1 improves by 15.5% over RAG.
Source: §1 · p. 3, §8 · p. 18Takeaways & Limitations
Security guidance is most useful when grounded in required task behavior and checked alongside functionality; repair feedback primarily recovers functionality while security tests check that tested protections remain.
Source: §8 · p. 18Generated test suites may be incomplete or incorrect, so they guide repair only; reported outcomes are independently determined by CWEval.
Source: §7 · p. 18
Abstract
from arXiv · showhide
Large Language Models (LLMs) can generate functionally correct code that remains insecure. Security-focused fine-tuning can be expensive and is not always available, motivating inference-time approaches. Security-oriented prompting and retrieval can reduce vulnerabilities, but may also remove required behavior or apply defenses incompatible with the task. The challenge is to identify relevant security knowledge and use it to improve security without sacrificing functionality. We present SecGent, an inference-time framework that retrieves weakness context from the Common Weakness Enumeration (CWE) database and applies it to secure code generation. By connecting task specifications to relevant textual CWE records, SecGent provides task-specific security guidance without relying on retrieved vulnerable or patched code examples. Complementary functional and security tests check the resulting implementation, and their execution feedback guides repairs toward satisfying both objectives. We evaluate SecGent with four open-source LLMs and Claude Sonnet 5.5 across five programming languages on the CWEval benchmark. It achieves the highest joint functional--security success against four inference-time baselines for all five models. Across the open-weight models, mean functional-security@1 (FS@1) improves by 15.5% over RAG, with gains in both functionality and security, and by 19.8% over the strongest prompt optimizer compiled for each model. On Sonnet, FS@1 reaches 92.1%, outperforming the strongest baseline AutoGen by 4.8%. In a separate comparison, SecGent outperforms Claude Code with security-focused prompting by over 6% in FS@1. Feedback-guided repair further improves FS@1 by up to 6.7% over generation without feedback, primarily by recovering functionality while checking security.
1 Introduction
LLMs can produce functionally correct but insecure code, while security interventions can remove required behavior. SecGent retrieves task-relevant CWE records and uses functional and security tests to guide generation and repair.
Motivation
- Passing ordinary tests does not establish security, and interventions can reject valid inputs, remove required operations, or apply incompatible defenses.Secure code generation must address relevant weaknesses without losing required behavior.Source: §1 · pp. 1–2
- Direct prompting can pass functional tests while leaving security failures undetected, and security-focused retrieval can improve security while rejecting valid behavior.For the header task, RAG rejects valid language lists; four of its five samples fail the functional oracle, while 29.2% of functionally correct Direct Prompting samples across four open-weight models fail the security oracle.Source: §1 · p. 2
- Task specifications describe intended behavior but may omit weakness cues, making relevant CWE retrieval and task-specific interpretation distinct challenges.For example, the header task does not mention carriage returns, newlines, or response splitting.Source: §1 · p. 2
SecGent overview
- SecGent retrieves task-relevant textual CWE records to guide implementation, derives complementary functional and security tests, and uses execution failures to guide bounded repair.Repairs are retained only when they improve one generated-test objective without reducing the other; the coding agent receives no vulnerable or patched code examples.Source: §1 · p. 2
Contributions
- Across open-weight models, mean FS@1 improves by 15.5% over RAG and 19.8% over the best prompt optimizer for each model.SecGent also outperforms four inference-time baselines in FS@k across all evaluated open-weight and proprietary models.Source: §1 · p. 3
2 Background and Related Work
Prior work studies vulnerability detection, secure generation, retrieval, and execution feedback. SecGent combines task-specific CWE retrieval with executable functional and security checks to guide repair.
Security of LLM-generated code
- Security benchmarks cover vulnerability detection and secure generation, including CodeSecEval’s Python-only scope and CWEval’s executable joint assessment.SecVulEval focuses on C/C++ vulnerability detection, while SecurityEval and CyberSecEval evaluate vulnerability-oriented or security-sensitive generation tasks.Source: §2 · pp. 3–4
Security evaluation
- CWEval assesses functionality and security on the same candidate, because security checks alone can reward code that omits required behavior.The paper uses joint functional–security success as its primary outcome and reports both constituent metrics to expose trade-offs.Source: §2 · p. 4
Secure generation and prompt optimization
- SecGent retrieves task-specific weakness context from textual CWE records without model adaptation or benchmark-specific prompt compilation, then uses generated tests to check behavior and guide repair.This contrasts with model adaptation, constrained decoding, and prompt-optimization approaches discussed in prior work.Source: §2 · p. 4
Task-aware security retrieval
- Unlike CodeGuarder and RESCUE, SecGent does not retrieve vulnerable or patched code examples; it uses CWE records and task-conditioned retrieval to guide generation and security-test selection.The testing agent grounds the selected weakness in task inputs and observable behavior without seeing the candidate.Source: §2 · p. 4
Execution feedback and security testing
- Prior work uses execution feedback for debugging and repair, while SecGent freezes tests before generation and accepts repairs only when neither tested objective regresses.The paper could not include SecAwareCoder as a baseline because its replication package was unavailable at the time of writing.Source: §2 · p. 4
3 Methodology
SecGent builds a frozen CWE catalogue, retrieves task-relevant weakness context, and uses it to generate independent functional and security tests before code generation. Execution feedback guides repair, with a strict rule for retaining candidates.
3.1 Design Overview
- The workflow builds a fixed catalogue, retrieves task-relevant weaknesses, freezes functional and security tests before generation, and retains repairs only when tested objectives do not regress.The coding agent generates without seeing the tests; the same agent later proposes repairs based on execution diagnostics.Source: §3.1 · pp. 4–5
3.2 Weakness Catalogue Construction
- The offline catalogue transforms 944 MITRE CWE Research Concepts records into frozen retrieval entries with generated code-level descriptions and keywords.The LLM rewrite uses only each record’s name and description; the resulting entries retain the CWE identifier, name, and mitigations.Source: §3.2 · p. 5, Figure 3 · p. 5
3.3 Task-Conditioned Weakness Context
- HyDE query generation expresses plausible insecure implementations in weakness-oriented vocabulary, then dense retrieval and reranking select the task’s ordered CWE context.The reranker scores 20 retrieved records under three orderings and averages the scores; the five highest-scoring records guide generation and testing.Source: §3.3 · pp. 5–6
3.4 Test Generation
- Before code exists, the testing agent creates functional tests for required behavior and CWE-aware security tests for adversarial inputs and observable harmful effects.The agent does not see candidate implementations or their execution outputs, and a validated test suite is frozen for generation and repairs.Source: §3.4 · pp. 6–7
3.5 Coding Agent: Initial Generation
- The coding agent receives the task and retrieved context, but not the frozen tests, and is instructed to preserve the required interface and valid-input behavior.The initial candidate can still exhibit the retrieved weakness; the example preserves embedded newline and tab characters in a log message.Source: §3.5 · p. 7
3.6 Test Execution
- A language-specific execution adapter runs each candidate against the frozen suite in an isolated container and classifies candidate evidence, complete passes, or unusable suites.In the example, execution identifies a security failure while confirming ordinary-message behavior passes, giving repair a behavior to preserve.Source: §3.6 · p. 7
3.7 Evidence-Guided Repair and Retention
- Repair uses functional–security outcomes and bounded diagnostics; a proposal replaces the retained candidate only if it preserves every passed category and fixes at least one failure.Ties and regressions are rejected, and repair stops at a full pass or when the repair budget is exhausted.Source: §3.7 · p. 8
4 Experiments
The experiments compare SecGent with inference-time baselines and prompt optimizers on CWEval. They examine effectiveness, language variation, and generated-test feedback using joint and component success metrics.
4.1 Research Questions
- The study asks whether SecGent improves joint functional and security success over inference-time baselines and prompt optimizers, across languages and with generated-test repair.Its four research questions cover baseline effectiveness, prompt optimizers, language-specific effectiveness, and feedback-guided repair.Source: §4.1 · p. 8
4.2 Evaluation Benchmark
- CWEval contains 119 coding tasks across 31 CWE classes and five languages: C, C++, Go, JavaScript, and Python.The task counts are 31 C, 21 C++, 19 Go, 23 JavaScript, and 25 Python tasks.Source: §4.2 · p. 9
4.3 Baselines
- Baselines include Direct Prompting, RAG, CodeGuarder, AutoGen, and three prompt optimizers trained on a task partition and applied unchanged to held-out tasks.AutoGen uses coder–reviewer cycles, while SecGent uses executable functional and security feedback and a non-regression retention rule.Source: §4.3 · p. 9
4.4 Experimental Setup
- The evaluation covers four open-weight models and Claude Sonnet 5.5, with five independently sampled candidates per task and at most two SecGent test-generation attempts and repairs.Prompt optimizers are evaluated only on open-weight models because compiling them requires repeated generation and evaluation on training tasks, adding substantial API cost for Sonnet.Source: §4.4 · pp. 9–10
4.5 Metrics
- The study reports functional@k, secure@k, and joint functional–security success FS@k, with FS@k as the primary outcome.The tables report F@1, S@1, FS@1, and FS@5; constituent metrics show whether security gains accompany functional losses.Source: §4.5 · pp. 10–11
5 Results
Across inference-time baselines and prompt optimizers, SecGent achieves the strongest joint functional–security results. Its advantage spans languages, while generated-test feedback further improves results, chiefly by restoring functionality.
5.1 RQ1: Comparison with Inference-Time Baselines
- SecGent records the highest FS@1 for all five models; across four open-weight models, mean FS@1 is 71.6%, versus 61.9% for RAG and 59.1% for AutoGen.It also retains the highest FS@2 and FS@5 for all five models.Source: §5.1 · p. 11, Table 1 · p. 12
5.2 RQ2: Comparison with Prompt Optimizers
- SecGent reaches 73.1% mean FS@1, exceeding the 61.0% mean of the strongest prompt optimizer selected for each open-weight model.SecGent also leads each model at FS@1 in both functional and security success.Source: Table 2 · p. 12, §5.2 · pp. 12–13
5.3 RQ3: Language-Specific Effectiveness
- SecGent ranks first or second in 19 of 20 open-weight language–model comparisons and leads overall on every model.Prompt optimizers can improve one language while weakening another; language-specific results can also hinge on a single task in these small test sets.Source: Figure 5 · p. 13, §5.3 · pp. 13–14
5.4 RQ4: Impact of Generated-Test Feedback
- Repair raises mean FS@1 by 2.6% on open-weight models and 3.0% on Sonnet-5.5; 66 of 71 joint improvements restore functionality.Generated-test expectations can also cause regressions: the repair gate checks improvement against generated tests, and seven of 11 joint regressions occur with GPT-OSS.Source: §5.4 · pp. 14–16, Table 4 · p. 16
6 Discussion
SecGent is compared with Claude Code and examined for generated-test reliability and retrieval quality. The discussion finds that generic security prompting trades functionality for security, generated tests can misdirect repairs, and related weakness context can still help when the benchmark-labeled CWE is missed.
6.1 Comparison with an Agentic Coding Harness
- SecGent improves FS@1 by 6.2% over Claude Code’s stronger configuration while maintaining comparable functionality and achieving the highest S@1 and FS@k at every k.Claude Code’s functionality-focused configuration reaches F@1 of 94.2% but S@1 of 87.1%; security-focused prompting raises S@1 to 91.2% while lowering F@1 to 88.8%.Source: Table 5 · p. 16, §6.1 · pp. 16–17
6.2 Reliability of Generated Tests
- Among 11 joint regressions in 291 retained repairs, generated tests included incorrect functional expectations, security assertions rewarding unsafe behavior, and incomplete functional coverage.One incorrect `ls -l` expectation caused a repair to fabricate a directory entry; the Pareto gate retained it because both generated tests passed, although CWEval functionality failed.Source: §6.2 · p. 17, Figure 7 · p. 17
6.3 Retrieval Quality and Its Limits
- SecGent’s rewritten catalogue and HyDE query raise top-five CWE recall from 47.1% for direct retrieval to 89.1% after reranking across 119 tasks.Even on the 13 tasks where the benchmark-named weakness was not retrieved, 64.6% of SecGent programs passed security tests, compared with 47.7% for Direct Prompting.Source: §6.3 · pp. 17–18
- Irrelevant path-traversal context coincided with a decline from 15 to 4 functional samples out of 20 on a JavaScript task, although the comparison does not isolate retrieval’s effect.The discussion cautions that retrieved security guidance should fit the task while preserving specified behavior.Source: §6.3 · p. 18
7 Threats to Validity
The evaluation controls stochastic generation within comparisons and uses CWEval, rather than generated tests, to determine reported outcomes. Its evidence is limited to the evaluated models and function-level benchmark tasks.
Internal Validity
- Generated test suites may be incomplete or incorrect, so they guide repair only; reported functional and security outcomes are determined independently by CWEval.Unusable generated suites trigger no repair, and the initial candidate is retained.Source: §7 · p. 18
- Five candidates per task and fixed model, decoding, sample-count, and split settings mitigate stochasticity within each comparison, but do not remove it.The authors note that generation and test execution are stochastic and may affect individual outcomes.Source: §7 · p. 18
External Validity
- Evaluation covers four open-weight models and Claude Sonnet 5.5 on 119 CWEval tasks, but does not establish generalization across proprietary model families or repository-scale development.Prompt optimizers are evaluated only on open-weight models, and the benchmark uses function-level tasks rather than long-context or repository-scale settings.Source: §7 · p. 18
8 Conclusion
SecGent connects programming tasks to relevant CWE knowledge for generation, testing, and repair. Across five models and five languages, it achieves the highest joint functional–security success among evaluated inference-time baselines, and the authors identify test reliability and broader evaluation as future work.
8 Conclusion
- SecGent connects programming tasks to relevant CWE knowledge for generation, testing, and repair.The conclusion says security guidance is most useful when grounded in required task behavior and checked alongside functionality.Source: §8 · p. 18
- Across five models and five languages, SecGent achieves the highest joint functional–security success among evaluated inference-time baselines, including Claude Code.Execution feedback primarily recovers functionality, while security tests check whether repairs preserve tested protections; repair reliability depends on generated-test correctness and coverage.Source: §8 · p. 18
- The authors propose extending weakness sources, validating generated security tests against vulnerable and corrected variants, broadening valid-input coverage, and evaluating repository-scale development.Source: §8 · p. 18