MACGen: Toward Functionally Correct and Secure Code Generation via Multi-Agent Collaboration

• Secure code generation requires security decisions within implementation while preserving intended behavior, and security-oriented prompting can reduce vulnerabilities but compromise functional correctness.Relevant threats are often underspecified, so models must infer them and translate them into concrete, functionally consistent code choices.Source: §1 · pp. 1–2
• Retrieved guidelines and examples narrow the security-knowledge gap, but generic advice still leaves generators to adapt it to task-specific code context.Examples are language-specific and costly to collect, limiting practical coverage.Source: §1 · p. 2
• Shared-dialogue critique can blur agent roles and increase token cost, latency, and long-context degradation as dialogue grows.Le et al. (2024) assign safety and helpfulness concerns to distinct critics, but critics observing one another’s reasoning may drift from their assigned perspectives.Source: Figure 1 · p. 1, §1 · p. 2
• MACGen decomposes secure code generation into planning, security analysis, synthesis, and refinement, with dedicated agents exchanging structured artifacts rather than full dialogue histories.The security advisor generates task-specific CWE guidelines, and the reviewer routes perspective-separated feedback to the coder for targeted refinement.Source: §1 · pp. 2–3
• MACGen improves F&S@1 over direct prompting by 19.61 pp on average on CWEval, and by 9.95 pp with GPT-4o and 11.18 pp with GPT-4o-mini on BaxBench.CWEval covers six LLMs and five programming languages; BaxBench covers six backend languages.Source: §1 · p. 2

2.1 Multi-Agents for Code Generation

• Planning-based multi-agent code-generation work includes CODESIM, which refines step-by-step plans through input/output simulation, while MACGen applies planning to secure code generation (Islam et al., 2025).Planning and human problem-solving approaches have not been fully utilized in secure code generation; other multi-agent strategies improve code quality through structured task decomposition (Li et al., 2023; Hong et al., 2024; Qian et al., 2023; Islam et al., 2024, 2025).Source: §2.1 · p. 3

2.2 Secure Code Generation

• Inference-time security methods use prompting/reflection, injected knowledge, or agentic critique, but treat security as added knowledge or post-generation feedback rather than throughout development (Nazzal et al., 2024; Tony et al., 2025a; Bruni et al., 2025; Wang et al., 2025; Zhang et al., 2024; Tony et al., 2025b; Lin et al., 2025; Shi and Zhang, 2026; Nunez et al., 2024; Le et al., 2024).Model-adaptation approaches improve security through tuning, localized optimization, or internal feature control, but require access to model parameters or internals (He and Vechev, 2023; He et al., 2024; Li et al., 2024; Hasan et al., 2025; Huang et al., 2026; El Husseini et al., 2026). Tony et al. (2025a) report that multi-turn Recursive Criticism and Improvement reduces security weaknesses.Source: §2.2 · p. 3

3.2 Overall Workflow

• The four-agent workflow uses a Planner, Security Advisor, Code Generator, and Reviewer to move from functional planning and draft code through security analysis, final generation, and iterative verification.The Security Advisor can exit early or use a security knowledge base to produce guidance before final code generation.Source: §3.2 · p. 3, Figure 2 · p. 4

3.3 Building a Standards-driven Security Knowledge Base

• The standards-grounded knowledge base draws on CERT and OWASP, structures guidelines as WHAT/WHY/HOW/EXAMPLE records, and retrieves top-K cosine matches for CWE-derived queries.Retrieved guidelines from the task-specific queries are combined into a candidate set.Source: §3.3 · pp. 3–4

3.4.1 Planner

• The Planner creates a concise functional plan that guides implementation and gives the Security Advisor task-specific context for identifying attack surfaces.The plan serves both functional correctness and security analysis.Source: §3.4.1 · p. 4

3.4.2 Security Advisor

• The Security Advisor triages for attack surfaces, exits early when it finds no risk, or uses plan- and CWE-guided retrieval followed by draft-code analysis to create compatible security guidelines.It combines guidelines from plan-level and code-level analyses, then checks them against functional requirements.Source: §3.4.2 · pp. 4–5

3.4.3 Code Generator

• The Code Generator first drafts code from the prompt and plan, then generates executable code under both functional-plan and security-guideline constraints and self-refines until syntactically valid.A syntax checker is used during the iterative self-refinement.Source: §3.4.3 · p. 5

3.4.4 Reviewer

• The Reviewer separately checks functionality and security, sends targeted feedback to the Code Generator, and repeats review until approval or the iteration limit.It checks for functional errors and residual security issues, such as missing input sanitization or risky functions.Source: §3.4.4 · p. 5

4.1 Benchmarks

• CWEval covers 119 code-completion tasks in five languages, while BaxBench covers 392 backend tasks across 28 scenarios, 14 frameworks, six languages, and single- and multi-file settings.Source: §4.1 · p. 5

4.2 Baselines & LLM Models

• MACGen is compared with five baselines across six LLMs.The baselines are Direct Prompting, SECGUIDE (Tony et al., 2025b), CODEGUARDER (Lin et al., 2025), RESCUE (Shi and Zhang, 2026), and INDICT (Le et al., 2024). The models are GPT-4o, GPT-4o-mini (Hurst et al., 2024), Gemini 2.5-Flash, Gemini 2.5-Flash-Lite (Comanici et al., 2025), DeepSeek-R1-Distill-Llama-70B (Guo et al., 2025), and Qwen3-8B (Yang et al., 2025).Source: §4.2 · p. 5

4.3 Implementation Details

• MACGen uses up to C = 3 potential CWE groups per CWEval task and C = 2 per BaxBench task, with M = 2 refinement iterations and decoding temperature 0.0.Baseline hyperparameters follow their original papers.Source: §4.3 · p. 5

4.4 Evaluation Metrics

• Both benchmarks use built-in test oracles to evaluate functionality and security; CWEval targets one CWE per task, while BaxBench uses end-to-end API-level exploits that may cover multiple CWE classes.The reported Pass@1 variants are Func@1 and F&S@1 over all generations, and Sec@1 over compilable generations.Source: §4.4 · pp. 5–6

5.1 Performance on Secure and Functionally Correct Code Generation

• Up to 70.59% F&S@1, MACGen leads CWEval across six LLMs and achieves the highest BaxBench F&S@1 on GPT-4o and GPT-4o-mini.On CWEval, MACGen has an average Func@1 drop of −0.56%p and Sec@1 gain of +23.82%p; on BaxBench GPT-4o, it improves F&S@1 from 21.17% to 31.12% over Direct.Source: Table 2 · p. 6, §5.1 · pp. 6–7

5.2 Performance Across Different Programming Languages

• MACGen achieves the best F&S@1 across all five CWEval languages, with especially large improvements on C++ and Go.The Security Advisor translates generic security concepts into task-specific constraints, reducing the Code Generator’s burden of interpreting language-variable guidelines.Source: §5.2 · p. 7, Figure 4 · p. 7

5.3 Token Cost

• MACGen consumes only 16% of INDICT’s tokens, costs $0.09 more than SECGUIDE, and costs $3.77 less than INDICT.The comparison covers 25 Python tasks from CWEval; the paper attributes this efficiency to giving agents only relevant artifacts.Source: §5.3 · p. 7, Table 3 · p. 7

5.4 Effect of Artifact-Only Coordination

• Against MACGEN-Shared, MACGen improves by +20.17% on CWEval GPT-4o-mini and +8.67% on BaxBench GPT-4o.MACGEN-Shared gives each agent the full accumulated upstream context, including intermediate reasoning and scratchpads; the reported results support artifact-only context locality across benchmarks and models.Source: §5.4 · p. 7

6.1 Impact of Different Agents

• The full MACGEN configuration achieves the highest F&S@1: the Security Advisor gives the largest single-agent gain but trades functionality, the Reviewer partly offsets that drop while preserving high Sec@1, and the Planner yields the best balance.Source: §6.1 · pp. 7–8, Table 4 · p. 8

6.2 Effectiveness of Security Advisor Components

• Removing any Security Advisor component degrades F&S@1 across both models; either generation step’s removal favors security over functionality, while bypassing validation introduces instability.The ablations remove standards retrieval, draft-code analysis, or final consolidation, respectively.Source: §6.2 · p. 8, Table 5 · p. 8

6.4 Impact of Different Number of Samplings

• At k=3 and temperature 0.5 on CWEval, MACGEN outperforms all baselines across metrics, suggesting its workflow benefits from sampling diversity and finds a secure, functional solution within three attempts.Source: Table 7 · p. 8, §6.4 · p. 8

6.3 Performance on General Functional Tasks

• MACGEN maintains performance comparable to Direct on HumanEval and HumanEval+ across LLMs, with some improvements; GPT-4o’s high early-exit ratio supports the triage mechanism on non-security tasks.Source: §6.3 · p. 8, Table 6 · p. 8

• MACGEN decomposes secure code generation into planning, security analysis, code synthesis, and refinement to generate functionally correct and secure code.Source: §7 · p. 8
• Specialized agents use artifact-only interfaces to prevent context bloat and preserve role clarity.Source: §7 · p. 9
• Experiments across six LLMs and eight programming languages demonstrate improvements in both functionality and security.Source: §7 · p. 9

Evaluation Granularity

• Security hardening can satisfy best practices yet break functionality under oracles expecting specific API behavior, making security scope difficult to evaluate.The evaluation follows each benchmark’s security scope and functional oracles; graded hardening levels or tests parameterized by security assumptions are left for future work.Source: Limitations · p. 10

Reasoning-Centric Design

• MACGEN delegates synthesis, analysis, and refinement to agent reasoning, using retrieval only to bridge security knowledge gaps; external verification remains complementary.Static analyzers and automated penetration testing could further strengthen security guarantees.Source: Limitations · p. 10

Ethical considerations

• MACGEN aims to mitigate security risks from vulnerable LLM-generated code and promote responsible, safety-aware deployment in software engineering.Evaluations use public academic benchmarks and open-source materials in controlled, offline settings; the authors report residual risks and limitations and plan to release code, scripts, prompts, and guideline-generation procedures.Source: Ethical considerations · p. 10

A Experiments Setup

• The experiments rely on publicly available tools and datasets, and LLM APIs are used under their official terms of service.The listed resources include HumanEval, LLMSecEval, CWEval, BaxBench, CodeQL, and the Insecure Code Detector.Source: A Experiments Setup · p. 14

B Evaluation Details

• CWEval tests functional correctness and a designated CWE trigger, whereas BaxBench executes complete backend applications and assesses functionality with end-to-end API tests and security with expert-written exploits.For BaxBench, the Code Generator also summarizes requirements, the functional plan, and security guidelines before generating code; F&S@k measures the chance that at least one of k attempts meets both criteria.Source: B Evaluation Details · p. 14

C Implementation Details

• In a manual sample of 100 normalized guidelines, major meaning distortions and genuinely incorrect advice each occurred in 3% of samples.The reported errors were concentrated in Go and CERT C++; the authors attribute them largely to sparse source material or language-specific nuances.Source: C Implementation Details · p. 15

D Security Guidelines

• MACGEN’s Security Advisor turns retrieved guidelines into task-specific constraints aligned with the target code, unlike the more general-purpose guidance from SECGUIDE, CODEGUARDER, and RESCUE.The examples describe MACGEN guidance as addressing concrete mitigations in the task’s implementation context rather than leaving the generator to adapt generic advice.Source: D Security Guidelines · p. 15

E Impact of Guideline Synthesis and Knowledge-Base Refinement

• Table 8’s ablations indicate that raw-knowledge injection weakens F&S performance, while MACGEN achieves the highest F&S@1 scores.Guideline synthesis without knowledge-base refinement can improve security metrics in some cases, particularly for GPT-4o-mini, but may reduce functional correctness; MACGEN combines synthesis with refined, structured knowledge.Source: Table 8 · p. 16, E Impact of Guideline Synthesis and… · p. 16

F Performance with the maximum number of inferred CWE groups C

• C=10 achieves the highest F&S@1, but inferred CWE groups average 1.01, 2.33, 2.95, and 3.51 for C=1, 3, 5, and 10, respectively, motivating default C=3.The experiment excludes the post-generation Reviewer; the authors select C=3 to capture the effective reasoning range while maintaining cost efficiency.Source: Table 9 · p. 16, F Performance with the maximum number… · p. 16

G Analysis of Security Error

• MACGen reduces security failures across many of the Top-15 CWE categories compared with GPT-4o Direct prompting.The categories include input validation and path traversal, injection-style vulnerabilities, cryptographic misuse, and SSRF or query-construction errors.Source: G Analysis of Security Error · pp. 16–17, Figure 7 · p. 17
• Residual failures cluster around API-specific knowledge gaps and weak verification of whether mitigations are semantically sufficient.CWE-327, CWE-329, and CWE-643 can require runtime-grounded API semantics; CWE-117 and CWE-347 expose insufficient checks for newline-based log injection, benchmark-specific timestamp formatting, and exact algorithm pinning. Suggested directions include runtime-grounded API knowledge, oracle-aware validation, and stricter reviewer criteria.Source: G Analysis of Security Error · p. 17
• 6.7% and 6.9% not-secure rates on early-exit samples for GPT-4o and Gemini 2.5 Flash, respectively, fall to 1.7% across the full benchmark.The passage says these rates confirm that the advisor reliably identifies low-risk tasks for these models.Source: G Analysis of Security Error · p. 17, Table 10 · p. 17
• 23.1% and 11.8% not-secure rates on exited samples for GPT-4o-mini and Gemini 2.5 Flash-lite indicate greater difficulty distinguishing security-sensitive functions from algorithmic tasks.On Back-Bench, GPT-4o triggers early exit on only 2 samples and GPT-4o-mini on none, reflecting robust routing of security-intensive tasks to full analysis.Source: G Analysis of Security Error · p. 17

H Language-specific analysis

• MACGEN gains across languages, but its improvement margin is relatively modest in JavaScript, mainly because of rigid evaluation oracles and library-specific nuances.Manual inspection attributes the margin to evaluation and implementation issues rather than a deficiency in MACGEN’s security reasoning.Source: H Language-specific analysis · p. 17
• For CWE-327-1, MACGen’s bcrypt replacement is marked a failure because the benchmark accepts only argon2 or hashlib outputs.The passage describes bcrypt as a widely accepted and still-secure password hashing function.Source: H Language-specific analysis · p. 17
• For CWE-020-0, a semantically equivalent URL with a trailing slash is marked a functional failure because it differs from the expected string.MACGen produces “https //music.example.com/” rather than the expected string without the slash.Source: H Language-specific analysis · p. 17

I Additional Results for Effect of Artifact-Only Coordination

• MACGEN-Shared keeps the four-agent pipeline but gives the Planner, Security Advisor, and Code Generator accumulated upstream context instead of structured artifacts alone.This variant differs from MACGEN solely in its coordination interface.Source: I Additional Results for Effect of… · p. 18
• In MACGEN-Shared, the Reviewer receives no upstream context and performs functional and security analysis in one accumulated session rather than separate independent passes.Source: I Additional Results for Effect of… · p. 18
• Tables 11 and 12 present the full numerical comparison of MACGEN and MACGEN-Shared on CWEval and BaxBench, respectively.The supplied passages identify these tables but provide no numerical values.Source: I Additional Results for Effect of… · p. 18, Table 11 · p. 18, Table 12 · p. 18

J Additional Evaluation on LLMSecEval

• LLMSecEval covers 150 Python and C tasks without functional test oracles, making this a security-only static-analysis evaluation.The evaluation reports Sec@1 for compilable generations, labeling outputs vulnerable when either CodeQL or ICD finds a CWE-specific issue and secure only when neither does; it also reports non-compilable counts (#NC), where lower is better.Source: J Additional Evaluation on LLMSecEval · p. 18
• MACGEN achieves the highest Sec@1 across all five evaluated models and maintains low non-compilable output counts.The gains are especially pronounced for GPT-4o-mini and Gemini 2.5 Flash-Lite, exceeding direct prompting and prior security-oriented baselines.Source: Table 14 · p. 18, J Additional Evaluation on LLMSecEval · p. 18

K Prompts

• The prompts omit the planning agent prompt, which is reused from Islam et al. (2025), and refine raw security text into reusable plain-text guidelines.The extraction prompt specifies a target language and requests only the guideline text, or an empty string when no security content is present.Source: K Prompts · pp. 18–19
• Security Advisor prompts check draft-code attack surfaces, identify likely cause–effect CWE groups, generate task-specific guidelines, then validate and refine them against functional requirements.They also generate lightweight, code-evident guidelines from the draft code.Source: Figure 8 · p. 19, Figure 9 · p. 20
• Code Generator prompts use the functional plan and security guidelines to generate secure code, refine it from evaluation feedback, and optionally pre-comprehend complex tasks..Source: Figure 10 · p. 21
• Reviewer prompts separately assess functional correctness and security against generated guidelines, then convert unsatisfied feedback into minimal actionable fixes.The prompt set includes distinct functional and security analyses of the current code.Source: Figure 11 · p. 22, K Prompts · p. 22