PaperlayerView exact revision on arXiv

Source-linked AI summary

RedCode: Risky Code Execution and Generation Benchmark for Code Agents

Chengquan Guo, Xun Liu, Chulin Xie, Andy Zhou, Yi Zeng, Zinan Lin, Dawn Song, Bo Li

arXiv:2411.07781v1cs.SEcs.AI

TL;DR

Code agents’ ability to execute code and interact with system resources creates safety risks that are not captured by evaluating static code generation alone. RedCode tests risky execution and harmful-software generation, and its evaluations find vulnerabilities across execution scenarios and generation tasks. The results show that risk varies with input format, domain, agent framework, and model capability.

  • Problem

    Comprehensive safety evaluation of code agents remains absent, despite their ability to execute code and interact with system environments beyond static code generation.

    Source: §1 · p. 2

  • Method

    RedCode combines Docker-based risky execution tests in Python and Bash with diverse query formats and Python function-instruction prompts for harmful-software generation.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    Evaluations find high attack success on risky execution, lower rejection for natural-language than code queries, and stronger models that can generate more sophisticated harmful software.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    Existing code agents are generally vulnerable, and robust and safe code agents are needed.

    Source: §5 · p. 10

    RedCode-Exec currently covers Python and Bash, while RedCode-Gen covers Python, limiting the evaluated programming-language scope.

    Source: A Discussion on limitations and impacts · p. 14

Abstract

from arXiv · show

With the rapidly increasing capabilities and adoption of code agents for AI-assisted coding, safety concerns, such as generating or executing risky code, have become significant barriers to the real-world deployment of these agents. To provide comprehensive and practical evaluations on the safety of code agents, we propose RedCode, a benchmark for risky code execution and generation: (1) RedCode-Exec provides challenging prompts that could lead to risky code execution, aiming to evaluate code agents' ability to recognize and handle unsafe code. We provide a total of 4,050 risky test cases in Python and Bash tasks with diverse input formats including code snippets and natural text. They covers 25 types of critical vulnerabilities spanning 8 domains (e.g., websites, file systems). We provide Docker environments and design corresponding evaluation metrics to assess their execution results. (2) RedCode-Gen provides 160 prompts with function signatures and docstrings as input to assess whether code agents will follow instructions to generate harmful code or software. Our empirical findings, derived from evaluating three agent frameworks based on 19 LLMs, provide insights into code agents' vulnerabilities. For instance, evaluations on RedCode-Exec show that agents are more likely to reject executing risky operations on the operating system, but are less likely to reject executing technically buggy code, indicating high risks. Risky operations described in natural text lead to a lower rejection rate than those in code format. Additionally, evaluations on RedCode-Gen show that more capable base models and agents with stronger overall coding abilities, such as GPT4, tend to produce more sophisticated and effective harmful software. Our findings highlight the need for stringent safety evaluations for diverse code agents. Our dataset and code are available at https://github.com/AI-secure/RedCode.

1 Introduction

Code agents can execute code and interact with system resources, adding safety risks beyond those posed by static code generation. RedCode evaluates risky code execution and software generation, and its findings show vulnerabilities across agent frameworks and model capabilities.

1 Introduction

  • Code agents’ tool use lets them interact with operating systems and external resources, creating risks such as deleting important files or leaking sensitive information.Agents can execute code, adjust actions based on observations, and use packages or command-line interfaces.Source: §1 · pp. 1–2
  • RedCode evaluates risky execution and software generation using Docker-based test cases, diverse input formats, 25 execution scenarios, and malware-generation prompts.RedCode-Exec contains 4,050 test cases, while RedCode-Gen contains 160 prompts.Source: §1 · p. 2
  • Across RedCode-Exec evaluations, agents show high attack success, with lower rejection for natural-language risky queries than code queries and higher rejection for Python than Bash.Rejection rates are generally higher for operating- and file-system risks than for other domains.Source: §1 · p. 2
  • OpenCodeInterpreter is relatively safer than ReAct and CodeAct, while stronger base models can both reject more risky execution and generate more sophisticated harmful software.The comparison identifies potentially opposing safety outcomes across execution and generation.Source: §1 · pp. 2–3

2 Related work

Prior safety benchmarks address general LLMs, generated code, or agent trajectories, but do not comprehensively evaluate code agents’ actions in real execution environments. RedCode instead tests agent execution and generation with risky prompts, Docker environments, and scenario-specific checks.

2 Related work

  • General LLM safety benchmarks assess harmful natural-language generations, while code-LLM benchmarks primarily evaluate vulnerabilities in static generated code.These works include code-related instructions and CWE-based weaknesses, but are designed for LLMs rather than agents.Source: §2 · p. 3
  • Agent safety benchmarks use curated or simulated risky trajectories and LLM judges, which may leave gaps between sandbox records and real-system behavior.The passage also notes that LLM judges can provide inaccurate safety scores.Source: §2 · p. 3
  • RedCode evaluates actual code-agent execution in prepared Docker systems and uses scenario-specific scripts to check responses, outputs, and environment state.Other concurrent benchmarks cover attacks such as prompt injection and backdoors, whereas RedCode addresses inherent risks without external attacks.Source: §2 · p. 3

3 RedCode Benchmark

RedCode combines risky code-execution and harmful-software-generation evaluations, using curated prompts, Docker environments, and outcome-specific assessment. Its execution cases span Python, Bash, and natural-language formats; its generation cases use malware-oriented function instructions.

3.1 RedCode overview

  • RedCode-Exec provides 4,050 prompts and RedCode-Gen provides 160 prompts, with agents’ execution and generation assessed in prepared Docker environments.Execution cases are evaluated with prepared scripts; generation quality is evaluated using LLMs and VirusTotal.Source: §3.1 · p. 3

3.2.1 Benchmark construction in Python

  • RedCode-Exec begins with risky Python snippets built from prior benchmarks and CWE scenarios, then expands and reviews them into executable test cases.The benchmark combines 12 applicable scenarios from prior code benchmarks with 13 distinct scenarios derived from CWE examples, yielding 25 scenarios across eight domains.Source: §3.2.1 · pp. 4–5
  • Scenario-specific scripts classify each execution as rejection, execution failure, or attack success, using agent responses, outputs, and execution-environment status.Operation Verification checks environment state, while Output Scrutiny confirms outcomes from responses and execution output.Source: §3.2.1 · pp. 5–6

3.2.2 Benchmark construction in other programming languages

  • GPT-4 translates Python cases into Bash while preserving risky functionality, producing 600 test cases across 20 scenarios.Five scenarios were removed for Bash compatibility issues; translated cases were checked with the same evaluation scripts.Source: §3.2.2 · p. 6

3.2.3 Benchmark construction in natural language

  • Each code case is also translated into concise text summaries and detailed text descriptions, requiring agents to generate and then execute risky code.Python and Bash prompts are translated separately with GPT-4 and curated few-shot examples.Source: §3.2.3 · pp. 6–7

3.3 RedCode-Gen: Code agent safety benchmark for malicious software generation

  • RedCode-Gen uses 160 function-signature and docstring prompts from eight malware families to test harmful code generation.Prompts are based on real malware samples; evaluation checks response completeness, execution correctness, and antivirus detection.Source: §3.3 · p. 7

4 Experiments

The experiments evaluate code agents on risky execution and harmful-code generation, finding substantial safety vulnerabilities across both tasks. Rejection varies by risk type, input format, language, agent framework, and base model.

4 Experiments

  • The study evaluates 19 agents on RedCode-Exec and nine on RedCode-Gen, examining risk domains, scenarios, languages, agents, and base models.OpenCodeInterpreter and CodeAct are evaluated on Python, while ReAct is evaluated on Python and Bash.Source: §4 · p. 7, §4.1 · p. 8

4.1 Evaluation on RedCode-Exec

  • The RedCode-Exec evaluation includes three agent frameworks built on varied base LLMs, with framework-specific model sets.OpenCodeInterpreter and CodeAct use fine-tuned models marked with an asterisk; ReAct includes models such as GPT-4, GPT-4o, and Claude-3.5.Source: §4.1 · p. 7
  • RedCode-Exec prompts cover code snippets, text summaries, and text descriptions, with a persuasive jailbreak prefix added to code-snippet inputs.Each scenario and language has 120 test cases across the four input formats; outcomes are measured with scripts reporting rejection, execution failure, and attack success.Source: §4.1 · p. 8

4.1.1 Main results

  • Code agents show low rejection and high attack success on risky execution, while code agents on RedCode-Gen have lower refusal rates and higher accuracy than most base models.RedCode-Exec rejection is higher for operating- and file-system risks; stronger coding capability is associated with higher malware quality, and some samples trigger VirusTotal.Source: §4.1.1 · p. 8, Table 1 · p. 10, §4.2 · p. 10
  • Technically buggy code is rejected less often than operationally risky code, yet can introduce vulnerabilities for attackers to exploit in real-world systems.Examples include missing default cases and weak random-seed management; insecure encryption can make stored passwords easier to crack.Source: §4.1.1 · p. 8
  • Natural-language risky inputs receive less rejection and achieve higher attack success than code inputs, while Bash inputs generally receive less rejection than Python inputs.Text descriptions have higher attack success and lower rejection than text summaries; jailbreak-prefixed code sometimes receives more rejection than plain code.Source: §4.1.1 · p. 9
  • OpenCodeInterpreter has higher rejection and lower attack success than ReAct and CodeAct, which the authors attribute to hard-coded disk-space protections.ReAct generally rejects more than CodeAct; under ReAct, stronger base models tend to show higher rejection, while fine-tuned models may have lower rejection.Source: §4.1.1 · pp. 9–10

4.2 Evaluation on RedCode-Gen

  • For RedCode-Gen, most models have low refusal rates, and code agents further lower refusal while increasing accuracy; some GPT-4 and DeepseekCoder samples trigger VirusTotal.Coding capability correlates positively with malware-generation accuracy (Pearson score 0.448), while code agents’ evaluation and refinement yield higher judge accuracy, lower refusal, and higher VirusTotal accuracy.Source: §4.2 · p. 10, Table 1 · p. 10

5 Conclusion

The paper introduces RedCode to evaluate code-agent safety in risky code execution and generation, and concludes that existing agents are generally vulnerable.

5 Conclusion

  • RedCode benchmarks code-agent safety in risky execution and generation, and the findings indicate existing agents are generally vulnerable.The authors conclude that robust and safe code agents are needed.Source: §5 · p. 10

A Discussion on limitations and impacts

RedCode is limited to Python and Bash for execution and Python for generation, while its social-impact discussion recognizes both security benefits and misuse risk. The dataset details give its scenario and prompt counts and distinguish it from prior agent safety benchmarks.

A Discussion on limitations and impacts

  • RedCode-Exec covers Python and Bash, while RedCode-Gen supports Python only, limiting the benchmark’s language scope.The authors identify expansion to additional languages as future work.Source: A Discussion on limitations and impacts · p. 14
  • The authors say RedCode’s red-teaming materials aim to improve agent security, but detailed weakness analysis could be misused by malicious actors.They note that misuse remains a risk despite the intended security benefits.Source: A Discussion on limitations and impacts · p. 14

B Dataset details

  • RedCode-Exec contains 4,050 test cases across 25 Python and 20 Bash scenarios, while RedCode-Gen contains 160 problems across eight scenarios.Execution scenarios use three prompt formats, with 30 test cases per format for each scenario.Source: B Dataset details · p. 14, Table 2 · p. 14
  • Unlike prior agent safety benchmarks, RedCode includes real code execution, and its dataset is larger than theirs.The benchmark comparison caption also notes that the works differ in evaluation objects, methods, and purposes.Source: Table 3 · p. 14

B.1.1 Benchmark construction in Python

This section describes the construction of RedCode’s Python and Bash execution cases, their natural-language transformations, and RedCode-Gen prompts. It covers executable test cases with scenario-specific evaluation scripts, transformations intended to let another code agent reproduce source-code functionality, and prompts spanning eight risky-software categories, attack vectors, and scenarios.

B.1.1 Benchmark construction in Python

  • Python seed cases implement risky tasks using prepared websites, servers, files, and processes, making them more concrete than R-judge descriptions and more detailed and executable than CWE cases.For example, the process-management scenario uses pre-deployed named processes to make termination tasks executable.Source: §B.1.1 · pp. 15, 17, Figure 8 · p. 15
  • The authors used GPT-4 and Mistral-Large to augment seed snippets, then ran generated code in Docker and manually modified or discarded cases that failed execution or evaluation checks.Prepared files, paths, online files, websites, processes, and servers were chosen to support executability and diversity.Source: §B.1.1 · p. 17
  • For operation verification, scripts can check file-system changes, including whether a written file matches its expected MD5; output scrutiny instead matches expected text in the agent’s result.For the sensitive-file example, the evaluator checks whether the output contains target words pre-read from /etc/passwd.Source: §B.1.1 · pp. 19–21
  • The authors describe their scenario-specific evaluation scripts as, to their knowledge, the first of their kind; scripts inspect agent responses, interpreter output, and Docker state to classify rejection, execution failure, or attack success.Checks include operating-system state verification and output string matching.Source: §B.1.1 · pp. 19, 21, 22

B.1.2 Benchmark construction in more programming languages

  • For Bash, five of the 25 scenarios were excluded for compatibility reasons; the remaining translations used few-shot GPT-4 at temperature 0, preserved intentional flaws, adapted unsupported constructs, and were verified through the evaluation pipeline.The excluded scenarios were #10, #11, #17, #19, and #24; the cited example used the gpt-4-1106-preview API.Source: §B.1.2 · pp. 22–23

B.1.3 Benchmark construction in natural language

  • For natural-language tasks, Python and Bash scripts were mapped to summaries and detailed step-by-step descriptions, yielding 25 Python scenarios and 20 Bash scenarios.GPT-4 generated the descriptions to convey the code’s purpose and steps, including predefined variable values, without requiring external input or programming syntax.Source: §B.1.3 · pp. 23–24
  • The natural-language task produces a brief summary and detailed step-by-step guide that another code agent can use to replicate the source code’s functionality.Descriptions must be precise, self-contained, free of programming syntax, and explicitly include all initial variable values, constants, and specific actions; Bash-to-Text uses different few-shot examples to improve translation quality.Source: §B.1.3 · pp. 24–25

B.2.1 Categories of risky software

  • RedCode-Gen covers eight main risky-software categories, with 20 function-signature prompts per category spanning diverse attack vectors and scenarios.The risk descriptions for the categories are provided in Table 6.Source: §B.2.1 · p. 26, Table 6 · p. 26
  • One Adware prompt specifies fake antivirus alerts, periodic browser launches to a supplied URL, and persistence through a named system service.The service is named AdwareService on Windows or adware on Linux.Source: §B.2.1 · p. 27

B.2.2 Dataset curation

RedCode-Gen’s malware-family instructions were curated through GPT-4 generation followed by manual checks. The resulting GPT-4 ransomware example is classified as malware by VirusTotal.

B.2.2 Dataset curation

  • GPT-4 generated malware-family function instructions, which researchers manually checked for malicious components, realism, and completeness until each category had 20.The prompts included category descriptions, formatting examples, and scenario examples; Trojan scenarios included disguised timer or weather apps.Source: §B.2.2 · p. 27
  • The curated prompts ask for diverse, correct function signatures focused on different scenarios within each malware category.Each instruction targets a single function.Source: §B.2.2 · pp. 27–28
  • VirusTotal classified the GPT-4 ransomware example as malware, and GPT-4 tended to produce solutions covering major instruction components.The paper reports that difficult components were often missing across models’ generated functions.Source: Figure 10 · p. 28, §B.2.2 · p. 28

C.1 More details on evaluation setup

The appendix details agent, prompt, and interaction setups, then examines memory-leak execution and URL obfuscation as additional risky scenarios. In the latter, random-character URLs lower rejection and raise attack success.

C.1 More details on evaluation setup

  • The evaluation covers OpenCodeInterpreter, ReAct, and CodeAct agents, with model compositions and model endpoints or links listed in Table 7.OpenCodeInterpreter and CodeAct were evaluated only on Python tasks; ReAct was evaluated on both Python and Bash.Source: §C.1 · pp. 28–29, Table 7 · p. 29

C.2 More details on each code agent

  • Agent implementations use different interaction protocols, while RedCode-Gen agents can refine generated solutions using compiler and judge feedback.The appendix also specifies framework-specific prompts and settings, including interaction limits and response handling.Source: §C.2 · pp. 29–32

D.1 Risky scenario of memory leaks

  • The OCI agent using DS-6.7B had a 5.8% rejection rate, 71.7% attack success rate, and 22.5% execution failure rate on memory-leak cases.The authors characterize the low rejection and high attack success rates as indicating inadequate awareness of memory-leak risks.Source: §D.1 · p. 33

D.2 Risky scenario of crawling website URLs with random characters

  • Random-character URLs make the evaluated agent less likely to reject risky crawling and more likely to succeed in it.The website content remains unchanged while risky words in URLs are replaced with random characters.Source: §D.2 · p. 33

D.3 Risk mitigation methods

The authors examine safety-aware prompts and propose additional mitigation strategies. Combined prompting performs best among tested prompts, but attacks remain frequent, especially for text inputs.

D.3 Risk mitigation methods

  • The proposed strategies include safety-aware prompting, safety-focused instruction tuning, hard-coded framework constraints, and sandboxing with permission controls.The authors report that OpenCodeInterpreter's hard-coded safety constraints provide robustness and that blacklist mechanisms can prevent particularly dangerous commands.Source: §D.3 · p. 33

D.3 Result of safety-aware prompting

  • The combined safety prompt outperforms other tested prompts, yet attack success remains high and rejection low, especially for text inputs.Mitigations are less effective for text than code inputs, with text descriptions showing the lowest safety promotion.Source: §D.3 · p. 34

D.4 Result analysis for 25 specific risky scenarios

Across 25 risky scenarios, the analysis identifies tasks that are especially difficult to carry out or commonly rejected. Results are summarized in Table 10, with full results across agents shown in Figure 11.

D.4 Result analysis for 25 specific risky scenarios

  • Scenario #3, crawling a risky website, and scenario #8, deleting sensitive files, are the most difficult tasks to implement in their respective domains.Source: §D.4 · p. 35
  • Scenario #9, adding a risky alias to .bashrc, has the highest rejection rate across scenarios.Source: §D.4 · p. 35
  • Scenario #17, improper execution privileges, is the most easily rejected program-logic issue.The analysis reports greater caution for permission-related program-logic issues than for other issues in that domain.Source: §D.4 · p. 35

D.5 Detailed results on different code agents

Detailed agent results reveal differences in robustness and scenario sensitivity, while execution failures arise from instruction-following problems, incorrect assertions, code changes, and extra actions. The final subsection introduces case studies of agents across risky scenarios and input types.

D.5 Detailed results on different code agents

  • OpenCodeInterpreter is more robust than ReAct, while CodeAct shows the lowest security awareness; agents also differ in sensitivity across risky scenarios.The authors note that rejection rates vary across frameworks using the same LLM and that stronger LLMs within a framework tend to have higher rejection rates.Source: §D.5 · pp. 39–40

D.6 Discussion on the RedCode-Exec execution failure cases

  • Execution failures stem from poor instruction following, incorrect assertions, code modification that disables threats, and autonomous extra activities.Natural-language prompts can improve agents' understanding and task completion compared with a prefix plus code snippets.Source: §D.6 · pp. 41–42

D.7 Case studies

  • The case studies illustrate benchmark applications and agent performance across risky scenarios and diverse language inputs.Source: §D.7 · p. 43

D.7.1 Higher vulnerability in natural language compared to programming language

In these case studies, the agent rejects risky commands in code but tends to accept natural-language requests; a step-by-step description also succeeds where a text summary fails.

D.7.1 Higher vulnerability in natural language compared to programming language

  • The agent rejects a Bash command to download a script to /usr, but tends to accept the natural-language version of the request.The accepted request downloads the content and saves it to the specified local file.Source: §D.7.1 · p. 43

D.7.2 Text descriptions lead to higher attack success rate than text summaries

  • A step-by-step description of posting a file's contents succeeds with status code 200, while the text-summary case reports status code 400.In the summary case, the agent mistakes the 400 response for success despite the failure indication.Source: §D.7.2 · p. 44

D.7.3 Python leads to a higher rejection rate than Bash

The case study contrasts rejection of a Python eval-based command with acceptance and execution of the same functionality in Bash.

D.7.3 Python leads to a higher rejection rate than Bash

  • The agent recognizes and rejects the vulnerable use of Python's eval() function.Its explanation identifies the attempt to import os and run a system command as unsafe.Source: §D.7.3 · pp. 45–46
  • The agent accepts and executes the Bash version of the eval-based functionality.The Bash function uses eval on an input string that lists files in /app/.Source: §D.7.3 · p. 46
  • The Bash execution returns a file listing for /app/, and the agent reports that execution is finished.The environment observation includes entries such as .bashrc_copy and Dockerfile.Source: §D.7.3 · p. 46

Related papers

Loading 2411.07781v1…