MACGen: Toward Functionally Correct and Secure Code Generation via Multi-Agent Collaboration

Miseon Yu; Jaehoon Choi; Younghan Lee; Yunheung Paek

Source-linked AI summary

MACGen: Toward Functionally Correct and Secure Code Generation via Multi-Agent Collaboration

Miseon Yu, Jaehoon Choi, Younghan Lee, Yunheung Paek

arXiv:2608.25457v1Aug 26, 2026cs.CRcs.AIcs.MA

TL;DR

LLMs often generate vulnerable code, while secure code generation must jointly satisfy functionality and security. MACGen coordinates planning, security analysis, synthesis, and refinement through specialized agents and artifact-only interfaces. It improves F&S@1 over direct prompting on CWEval and BaxBench.

Problem
Secure code generation must jointly satisfy functional correctness and security, while existing guideline retrieval and shared-dialogue feedback leave task-specific translation and role separation limited.
Method
MACGen uses a Planner, Security Advisor, Code Generator, and Reviewer that exchange structured artifacts for planning, task-specific security guidance, synthesis, and refinement.
Results
19.61 percentage points: MACGen improves F&S@1 over Direct Prompting on CWEval on average, while improving it by 10.57 percentage points on BaxBench on average.
Takeaways & Limitations
Artifact-only interfaces preserve role specialization while MACGen jointly improves functionality and security across the evaluated benchmarks, models, and languages.
Takeaways & Limitations
The evaluation follows benchmark-specific security scopes and functional oracles, leaving graded hardening levels and security-assumption-parameterized tests for future work.

Abstract

from arXiv · show

Despite their strong ability to generate code, large language models often fail to produce secure code, as their outputs frequently contain security vulnerabilities. Secure code generation is inherently challenging because it requires solving a multi-objective problem: functional correctness and security. Existing approaches address this challenge by injecting external security knowledge or by using agentic feedback and iterative refinement. However, guideline retrieval often leaves the generator to translate generic advice into task-specific secure implementations, while shared-dialogue multi-agent feedback can blur role boundaries and suffer from context bloat. We present MACGen, a multi-agent framework that integrates planning, security analysis, code synthesis and refinement to jointly optimize security and functionality. A planner constructs a step-by-step plan to satisfy functional requirements. A security advisor identifies likely CWEs and synthesizes task-specific guidelines, a coder then generates code grounded in these artifacts, and a reviewer issues perspective-separated feedback. Rather than sharing full dialogue histories, each agent receives only structured artifacts from upstream stages, enforcing role specialization and reducing uncontrolled context growth. On CWEval and BaxBench, MACGen improves F&S@1 over direct prompting by 19.61 and 10.57 percentage points (pp) on average, respectively.

1 Introduction

Secure code generation requires jointly satisfying functional correctness and security, but existing knowledge-injection and shared-dialogue approaches leave task-specific translation and role interference unresolved. MACGen addresses these issues through specialized stages and artifact-only coordination, improving joint performance on two benchmarks.

LLM-generated code often contains security vulnerabilities despite strong code-generation capabilities.
Secure code generation is a dual-objective problem because security decisions, such as validation, safer APIs, and permission checks, can affect intended functionality.
Retrieved guidelines are generic and leave generators to translate task-independent advice into implementations aware of variables, signatures, and control flow.
Shared-dialogue critics may observe one another’s reasoning, causing role interference, overlapping feedback, and drift from independent functional or security evaluation.
MACGen decomposes generation into planning, security analysis, code synthesis, and refinement, passing compact artifacts through role-specialized interfaces.
19.61 percentage points: MACGen improves F&S@1 over Direct Prompting on CWEval, averaging across six LLMs and five programming languages; BaxBench gains 9.95pp with GPT-4o and 11.18pp with GPT-4o-mini.

2 Related Works

Prior work improves secure code generation through model adaptation, inference-time prompting, external security knowledge, and agentic critique. These approaches differ in access requirements and coordination strategy, while MACGen integrates security throughout development.

Model-adaptation methods improve security through parameter or internal-feature changes but require access to model parameters or internals.
Inference-time prompting and reflection steer models toward safer code without requiring model adaptation.
External security knowledge methods inject secure examples or guidelines, but retrieved guidance remains generic and secure examples can be language-specific and costly.
Agentic critique assigns specialized agents to review security or both functional and security perspectives, while shared dialogue can blur role boundaries.
Prior inference-time approaches generally treat security as added knowledge or post-generation feedback rather than integrating it throughout development.

3 MACGEN Design

MACGen uses four specialized agents and artifact-based interfaces to transform task requirements into functionally constrained, security-guided code. Its Security Advisor combines standards retrieval, code analysis, validation, and early exit before reviewer-driven refinement.

Overall Workflow: MACGen comprises a Planner, Security Advisor, Code Generator, and Reviewer connected through a staged workflow.
Planner: The Planner creates a functional plan that guides draft generation and gives the Security Advisor task-specific context for identifying attack surfaces.
Security Advisor: The Security Advisor can early-exit low-risk tasks or perform deeper analysis when potential attack surfaces are detected.
Security Advisor: For deeper analysis, the advisor maps risks to CWE groups, retrieves relevant standards, and generates task-specific security guidelines.
Security Advisor: The advisor supplements standards-driven guidance with draft-code analysis, then consolidates and validates both guideline sets against functional requirements.
Generation and Review: The Code Generator produces draft and final code under functional-plan and security-guideline constraints, while the Reviewer performs separated functionality and security checks and requests revisions.

4 Experimental Setup

The evaluation uses CWEval and BaxBench across multiple languages and models, comparing MACGen with direct prompting, retrieval-based systems, and an internal-dialogue multi-agent baseline. Performance is measured by functional, security, and joint success metrics.

Benchmarks: CWEval contains 119 code-completion tasks across C, C++, Python, JavaScript, and Go.
Benchmarks: BaxBench contains 392 backend tasks spanning 28 scenarios, 14 frameworks, six languages, and single- and multi-file applications.
Baselines and Models: MACGen is compared with Direct Prompting, three retrieval-augmented methods, and INDICT, an internal-dialogue multi-agent framework.
Baselines and Models: The experiments use six LLMs and report common MACGen settings including up to C = 3 CWE groups on CWEval, C = 2 on BaxBench, and M = 2 refinement iterations.
Evaluation Metrics: Both benchmarks automatically evaluate functionality and security with built-in test oracles, while BaxBench uses end-to-end API-level exploits that may cover multiple CWE classes.
Evaluation Metrics: Func@1, Sec@1, and F&S@1 measure functional correctness, security among compilable generations, and joint functional-security success, respectively.

5 Experimental Results

MACGEN is evaluated across multiple models, languages, and benchmarks for jointly functional and secure code generation, with strong F&S@1 performance, cross-lingual gains, and a comparatively efficient artifact-based workflow.

MACGEN achieves the best overall CWEval performance across all six LLMs, reaching up to 70.59% F&S@1.
MACGEN achieves the highest F&S@1 on BaxBench for both GPT-4o and GPT-4o-mini.On GPT-4o, it improves F&S@1 from 21.17% to 31.12% over Direct; on GPT-4o-mini, it improves by 3.57 percentage points over INDICT.
MACGEN achieves the best F&S@1 across all five CWEval programming languages, with especially large improvements on C++ and Go.The comparison averages GPT-4o and GPT-4o-mini results.
MACGEN consumes only 16% of INDICT’s tokens while reducing costs by $3.77 compared to INDICT.Across 25 Python tasks from CWEval, its cost is only $0.09 higher than SECGUIDE.
MACGEN consistently outperforms MACGEN-Shared across benchmarks and models, including a +20.17% improvement for GPT-4o-mini on CWEval.The comparison tests artifact-only interfaces against accumulated upstream context.

6 Ablation Study

Ablation results show that MACGEN’s specialized agents and Security Advisor components contribute complementary benefits to joint functionality-security performance, while broader evaluations preserve general coding ability and sampling robustness.

6.1 Impact of Different Agents: The full MACGEN configuration achieves the highest F&S@1, with the Planner, Security Advisor, and Reviewer contributing distinct complementary benefits.The Security Advisor provides the largest single-agent gain, while the Reviewer and Planner improve the functionality-security balance.
6.2 Security Advisor Components: Removing any individual Security Advisor component reduces performance across both evaluated models.The ablations remove standards-driven generation, code-based generation, or final validation.
6.3 General Coding Capabilities: MACGEN maintains performance comparable to Direct on HumanEval and HumanEval+, with some models showing improvements.The high GPT-4o early-exit ratio supports the effectiveness of triage on non-security tasks.
6.4 Sampling Stability: At k = 3 with temperature 0.5, MACGEN outperforms all baselines across Func@k, Sec@k, and F&S@k.The evaluation uses CWEval with GPT-4o.

7 Conclusion

MACGEN decomposes secure code generation into specialized stages connected by artifact-only interfaces to generate code that is both functionally correct and secure.

MACGEN decomposes secure code generation into planning, security analysis, code synthesis, and refinement.
Specialized agents use artifact-only interfaces to prevent context bloat and preserve role clarity.Experiments across six LLMs and eight programming languages demonstrate improvements in functionality and security.

Limitations

MACGEN improves functionality and security but remains limited by inference cost, evaluation ambiguity, and dependence on reasoning without external verification.

Efficiency: The multi-agent architecture incurs additional inference cost compared with single-pass methods despite artifact-only interfaces and early-exit triage.The paper frames this overhead as a trade-off for stronger assurance.
Evaluation Granularity: Benchmark-defined security scopes and functional oracles may not cleanly distinguish necessary hardening from over-engineering.The authors identify graded hardening levels and functionality tests under varying security assumptions as open directions.
Reasoning-Centric Design: MACGEN’s reasoning-centric design leaves external verification tools such as static analyzers and automated penetration testing for future work.The framework uses retrieval to bridge security knowledge gaps while delegating synthesis, analysis, and refinement to agent reasoning.

Ethical considerations

The paper frames MACGEN as a preventive approach to vulnerabilities in LLM-generated code and reports evaluations using public benchmarks, datasets, tools, and isolated environments.

Purpose: MACGEN is intended to mitigate security risks from vulnerable LLM-generated code through preventive secure code generation.The stated purpose is responsible, safety-aware deployment in software engineering contexts.
Risk Mitigation: The evaluations use public academic benchmarks in isolated environments to minimize misuse risk.The paper states that the datasets and code used are open-source or publicly available.
Resources: The study relies on publicly available tools and datasets, including HumanEval, LLMSecEval, CWEval, BaxBench, CodeQL, and ICD.The listed resources are used under their respective licenses or official terms of service.
Evaluation: CWEval tests functional correctness and security separately, while BaxBench evaluates complete backend applications through end-to-end tests and expert-written exploits.These benchmarks operationalize joint secure-code performance through task-specific test oracles and isolated execution.
Knowledge Base: The knowledge base uses authoritative general and language-specific secure-coding standards, frozen as a static snapshot for reproducibility.Sources include OWASP ASVS, CERT standards, OpenSSF practices, and OWASP language-specific guidance.
Validation: Manual validation of 100 normalized guidelines found major meaning distortions and incorrect advice in 3% of samples each.The sample covered five sources, with 20 guidelines per source.
Responsible Design: MACGEN transforms retrieved security knowledge into task-specific constraints instead of passing generic guidelines directly to the generator.The synthesis is grounded in the actual coding context, including implementation structure and security risks.

G Analysis of Security Error

MACGEN reduces failures across common CWE categories and uses early-exit triage, but residual errors reflect missing API semantics, verification weaknesses, and benchmark-oracle mismatches.

CWE Error Patterns: MACGEN reduces security failures across common categories including input validation, path traversal, injection, cryptographic misuse, SSRF, and query-construction errors.The result is attributed to staged decomposition identifying task-specific risks that direct generation often overlooks.
Residual Failures: Residual failures often require library- or API-specific knowledge unavailable from high-level decomposition alone.Examples include mutable cryptographic APIs and hallucinated library interfaces.
Residual Failures: Some remaining errors arise from insufficient mitigation verification, such as confusing broad algorithm-family checks with exact algorithm pinning.Other cases involve early-exit decisions or mismatches between sanitization behavior and benchmark expectations.
Early-Exit Triage: GPT-4o and Gemini 2.5 Flash show not-secure rates of 6.7% and 6.9% on early-exit samples, versus 1.7% across the full benchmark.These figures indicate generally reliable low-risk-task triage for the strongest proprietary models.
Early-Exit Triage: GPT-4o-mini and Gemini 2.5 Flash-lite have higher not-secure rates on exited samples, at 23.1% and 11.8%, respectively.The paper associates these rates with greater difficulty distinguishing security-sensitive functions from purely algorithmic samples.

H Language-specific analysis

MACGEN improves performance across languages, but JavaScript gains are comparatively modest because rigid evaluation oracles and library-specific details can penalize semantically acceptable implementations.

Language Results: JavaScript shows a relatively modest improvement margin compared with other languages despite gains across all languages.Manual inspection attributes this pattern primarily to rigid evaluation oracles and library-specific implementation nuances.
Oracle Effects: A bcrypt implementation was marked insecure because the benchmark accepted only argon2 or hashlib for the tested task.The example illustrates how oracle-specific expectations can reject an otherwise accepted secure primitive.
Oracle Effects: A URL with a trailing slash was marked functionally incorrect even though the output was semantically equivalent to the expected URL.The benchmark compared against an exact expected string.
Variant Design: MACGEN preserves the same four-agent pipeline while changing only its coordination interface in the MACGEN-Shared variant.This isolates coordination structure as the variant’s distinguishing factor.

J Additional Evaluation on LLMSecEval

MACGen is evaluated on LLMSecEval as a security-only static-analysis task because the benchmark lacks functional test oracles. It achieves the highest Sec@1 across the evaluated models while maintaining relatively few non-compilable outputs.

Evaluation setup: LLMSecEval contains 150 Python and C programming tasks without functional test oracles, so the evaluation measures security rather than joint functionality and security.
Metrics: Sec@1 is computed over compilable generations, requiring both CodeQL and ICD to report no CWE-specific findings for a secure label.Non-compilable outputs are separately reported as #NC, where lower is better.
Results: MACGEN achieves the highest Sec@1 across all five evaluated models.The gains are especially pronounced for GPT-4o-mini and Gemini 2.5 Flash-Lite compared with direct prompting and prior security-oriented baselines.
Results: MACGEN maintains a low number of non-compilable outputs across models, indicating that its security gains do not come from producing fewer compilable programs.
Agent prompts: The appendix specifies separate prompts for guideline refinement, CWE extraction, security analysis, code analysis, code generation, and review.The security advisor refines guidelines, checks attack surfaces, extracts likely CWE groups, generates code-evident guidance, and validates consolidated guidelines; the reviewer produces actionable fixes.