Source-linked AI summary
SandboxEval: Towards Securing Test Environment for Untrusted Code
Rafiqul Rabin, Jesse Hostetler, Sean McGregor, Brett Weir, Nick Judd
TL;DR
Executing LLM-generated code can put assessment infrastructure at risk, while the security of researchers’ test environments has received limited attention. SandboxEval provides handcrafted tests for sandboxed code-execution environments. Applied to Dyff, it largely confirmed risk mitigations while identifying configuration details for review and further improvement.
Problem
LLM-generated code may contain security flaws, and researchers have rarely discussed how to test the security of their own assessment systems that execute it.
Source: §I · p. 1Method
SandboxEval is a manually crafted suite of real-world scenarios for testing the security and confidentiality of environments that execute untrusted code.
Source: §I · p. 1, §II · p. 2, §VI · p. 9Results
SandboxEval largely confirmed that the Dyff instance mitigated significant risks and was prepared for untrusted code, while tests identified configuration details for further investigation and improvement.
Source: §III · p. 5Takeaways & Limitations
The test results provided Dyff researchers with a basis to review deployment details and consider whether additional mitigations were necessary before executing untrusted code.
Source: §VI · p. 9The 51 scenarios represent common threats but do not cover the full spectrum of possible malicious activities.
Source: §V · p. 9
Abstract
from arXiv · showhide
While large language models (LLMs) are powerful assistants in programming tasks, they may also produce malicious code. Testing LLM-generated code therefore poses significant risks to assessment infrastructure tasked with executing untrusted code. To address these risks, this work focuses on evaluating the security and confidentiality properties of test environments, reducing the risk that LLM-generated code may compromise the assessment infrastructure. We introduce SandboxEval, a test suite featuring manually crafted test cases that simulate real-world safety scenarios for LLM assessment environments in the context of untrusted code execution. The suite evaluates vulnerabilities to sensitive information exposure, filesystem manipulation, external communication, and other potentially dangerous operations in the course of assessment activity. We demonstrate the utility of SandboxEval by deploying it on an open-source implementation of Dyff, an established AI assessment framework used to evaluate the safety of LLMs at scale. We show, first, that the test suite accurately describes limitations placed on an LLM operating under instructions to generate malicious code. Second, we show that the test results provide valuable insights for developers seeking to harden assessment infrastructure and identify risks associated with LLM execution activities.
I. INTRODUCTION
LLM-generated code can contain security flaws, and executing it in assessment environments risks compromising the systems that run it. SandboxEval addresses the limited attention to testing those environments by introducing handcrafted scenarios and demonstrating their use on Dyff.
I. INTRODUCTION
- Safety research has rarely discussed how researchers can test their own systems, even as researchers release infrastructure-agnostic datasets and may not know their host environment’s configuration.Researchers also sometimes execute code on personal laptops with few or no security features.Source: §I · p. 1
- SandboxEval is a manually crafted suite simulating real-world safety scenarios for LLM execution environments running untrusted code.It targets risks including sensitive-information exposure, filesystem manipulation, and external communication, and is intended for assessment’s scoring step.Source: §I · pp. 1–2
- The authors deployed SandboxEval on a running Dyff instance in a cloud environment to test the configuration’s intended security effects and identify areas needing further investigation.The tests probe Linux systems and can be implemented across a range of code-execution environments.Source: §I · pp. 1–2
II. SANDBOXEVAL TEST SUITE
SandboxEval is an adaptable resource for assessing whether sandboxed environments are suitable for untrusted code execution. Its 51 tests span security and confidentiality concerns, including exposure of system, directory, and metadata information.
II. SANDBOXEVAL TEST SUITE
- SandboxEval’s 51 tests were selected through interdisciplinary collaboration and address security and confidentiality concerns in untrusted-code execution.The process drew on literature, industry experience, best practices, and implementation choices; tests may be especially relevant in less-controlled environments.Source: §II · p. 2
1) Exposing Sensitive Information - System, Directory, and Metadata:
- The suite’s exhaustive parameter list supports least-privilege discussions and checks whether an environment conforms to configuration expectations, though some failures may not matter in a given deployment.The authors note that locale and timezone may be exposed for basic tasks, while environment variables could enable exposure of sensitive information.Source: §II · p. 2, 1) Exposing Sensitive Information -… · p. 2
- Tests cover system details, directory paths and recursive exploration, and file or directory metadata such as permissions, attributes, and environment variables.The system category has ten cases, directory hierarchy six, and metadata three.Source: 1) Exposing Sensitive Information -… · pp. 2, 4
2) Manipulating Structures, Contents, and Privileges of Filesystem:
SandboxEval probes whether sandbox environments control unauthorized filesystem changes across filesystem structure, file contents, and privileges. Some failures may indicate critical vulnerabilities depending on deployment context.
2) Manipulating Structures, Contents, and Privileges of Filesystem:
- Seven structure tests probe locating, creating, moving, copying, renaming, deleting, and compressing restricted or critical files and folders.Uncontrolled file creation could exhaust resources, while deleting critical files could cause data loss.Source: 2) Manipulating Structures, Contents,… · p. 4
- Six content tests check whether critical files can be listed as readable, writable, or executable and whether corresponding read, write, or execute actions can be performed.Source: 2) Manipulating Structures, Contents,… · p. 4
- Four privilege tests check whether ownership or access rights of critical files and folders can be altered.Two test ownership transfers to root or the current user, and two alter access rights.Source: 2) Manipulating Structures, Contents,… · p. 4
3) Initiating External Communications and Dangerous Operations:
In a running Dyff instance, SandboxEval largely confirmed that significant risks were mitigated while identifying configuration details for review and additional steps to improve the configuration. Its test scenarios include external communications and potentially dangerous operations.
3) Initiating External Communications and Dangerous Operations:
- Eight external-communication tests probe Ping, DNS, HTTP, FTP, SSH, cloud storage, email, and messaging; seven dangerous-operation tests simulate harmful system and resource actions.The dangerous-operation scenarios include seeking root access, altering or deleting system files, consuming resources, and disrupting services.Source: 3) Initiating External Communications… · p. 5
III. EXPERIMENTATION AND RESULTS
- The Dyff deployment serves as a demonstration of SandboxEval for evaluating sandbox security and confidentiality in untrusted-code execution.Dyff is an open-source AI assessment framework.Source: §III · p. 5
- SandboxEval largely confirmed that the Dyff instance mitigated significant risks and was prepared for untrusted code, while identifying additional configuration-improvement steps.Exposed environment variables warranted further investigation and prompted review of container orchestration configuration.Source: §III · p. 5
A. Experimental Design
The study evaluates manually written SandboxEval cases on Dyff and explores whether Code Llama variants can generate similar tests. Because generated cases often required correction, the analysis proceeded with handcrafted cases and cautious execution methods.
Test generation
- Researchers hand-wrote Python cases for each test description and continued with them after LLM-generated cases often proved unusable or difficult to assess.They tried CodeLlama-7B, CodeLlama-7B-Python, and CodeLlama-7B-Instruct; generated cases could be syntactically valid yet poorly structured for execution.Source: A. Experimental Design · pp. 5–6
Dyff
- Dyff runs untrusted code in Kubernetes Pods and containers with restricted Pod security, deny-all network policies, gVisor, and resource limits and timeouts.Its deployment configuration also sandboxes containers used for inference and scoring.Source: A. Experimental Design · p. 5
Test generation
- After removing explanatory text, generated code averaged 40.13% syntactic validity; CodeLlama-7B-Python reached 45.0%.Before removing explanatory text, average validity was 16.47%, and CodeLlama-7B-Python had a 27.8% success rate.Source: A. Experimental Design · p. 6
Test execution
- Execution was cautious: filesystem risks were assessed through access rights, while proxy operations indicated likely outcomes for network and dangerous-operation tests.The authors note that proxy actions indicate risk but do not guarantee actual failure.Source: A. Experimental Design · p. 6
B. Result Analysis
The section summarizes executed test cases in Dyff using three outcome statuses.
B. Result Analysis
- Table IV summarizes the overall status of test-case executions within Dyff.The statuses are Accessed, Denied, and Unknown.Source: B. Result Analysis · p. 6
- Accessed means the code executed successfully and returned the expected outcome.This status distinguishes successful execution from cases that did not yield the expected outcome.Source: B. Result Analysis · p. 6
- Denied covers cases that failed to return the expected outcome or encountered a permission-related exception; Unknown means execution status could not be determined.Source: B. Result Analysis · p. 6
1) Test results for exposing sensitive information:
SandboxEval identified exposed environment variables, including sensitive values, while the system-information and filesystem findings indicated confinement to the container and no easy path to host manipulation.
1) Test results for exposing sensitive information:
- Exposed environment-variable values included sensitive keys, IP addresses, and SHA256 hashes, prompting review of whether those variables needed to be available to the container.Source: 1) Test results for exposing sensitive… · p. 7
- Code could access a wide range of system information, but namespace confinement generally limited its leverage for escaping to the host.Untrusted code could not access sensor information, other users, or process identifiers beyond its own.Source: 1) Test results for exposing sensitive… · p. 7
- The directory results showed no easy path to manipulate the host system; accessible root-directory files and directories were restricted to resources and jobs in the users’ own namespaces.Source: 1) Test results for exposing sensitive… · p. 7
2) Test results for manipulating filesystems:
SandboxEval found filesystem access and permissions constrained in the Dyff instance, while its network and dangerous-operation tests identified blocked actions that succeeded on a research laptop.
2) Test results for manipulating filesystems:
- Within ten directory levels, exploration found 527 readable files, 5,901 writable files, and 7,122 executable files across 136, 1,272, and 2,407 directories, respectively.Source: 2) Test results for manipulating… · p. 7
- The root user’s home directory was inaccessible; users could write only in /proc and /tmp and could not modify protected files, directories, ownership, or access rights.Source: 2) Test results for manipulating… · p. 7
3) Test results for external communications:
- Dyff blocked tested external communication routes, including public URLs, DNS, HTTP, SSH, FTP, and cloud-storage credentials; the same remote-control and exfiltration vectors succeeded on a research laptop.The results prompted review of network configuration policies, including whitelists.Source: 3) Test results for external… · p. 7
4) Test results for dangerous operations:
- Dyff’s resource limits, blocked outgoing traffic, and non-root execution thwarted tested resource misuse and disruptive operations that succeeded on a research laptop.The tests also probed how misconfigured environments could expose vulnerabilities under default settings.Source: 4) Test results for dangerous operations · pp. 7–8
IV. RELATED WORKS
Prior work examines the security of AI-generated code, including vulnerabilities in code produced by GitHub Copilot and ChatGPT. These studies motivate secure execution of generated code, but address the generated code’s security rather than the security of assessment infrastructure.
IV. RELATED WORKS
A. Security Issues in AI-Generated Code
- Across studies, GitHub Copilot outputs were found vulnerable in approximately 40% of programs, insecure in up to 27.25% of cases, and security-issue-prone in 32.8% of snippets.The studies are by Pearce et al., Majdinasab et al., and Fu et al. [32], respectively.Source: A. Security Issues in AI-Generated Code · p. 8
- Asare et al. found Copilot replicated vulnerabilities from original human-authored code about 33% of the time.Dakhel et al. also found Copilot suggestions contain more bugs than human developers’ suggestions.Source: A. Security Issues in AI-Generated Code · p. 8
- ChatGPT studies report insecure code, quality and CWE issues in modified user code, and potential issues in approximately one-third of generated Java and Python code.The cited studies include Khoury et al., Rabbi et al., and Liu et al..Source: A. Security Issues in AI-Generated Code · p. 8
B. Secure Execution of AI-Generated Code
Prior work uses sandboxing and isolation to reduce risks from executing untrusted code, while related evaluations often test whether an LLM generates attack code rather than whether assessment sandboxes are suitable. The paper also identifies limits to its test suite’s coverage and evaluation settings.
B. Secure Execution of AI-Generated Code
- Prior work uses isolated sandbox environments, including gVisor and Docker, to execute untrusted programs and reduce risks to host systems.Other cited studies likewise adopted sandboxing for execution of code from unverified sources [21] [45].Source: B. Secure Execution of AI-Generated Code · p. 8
- Sandbox-building research includes controlled execution architectures for LLM agents, per-plugin sandboxes, and simulated environments for agent tasks.The cited approaches also include an LLM emulator-and-evaluator framework [46] and a neural sandbox for classification.Source: B. Secure Execution of AI-Generated Code · p. 8
- Unlike evaluations of whether LLMs generate attack code or refuse sandbox-targeting requests, this work assesses whether assessment sandboxes can handle untrusted code.Related work also reported cross-session file access in an OpenAI GPT sandbox, with potential sensitive-information leakage [43].Source: B. Secure Execution of AI-Generated Code · pp. 8–9
- Reported infrastructure vulnerabilities include malicious package installation affecting Composer instances and an authentication flaw enabling file exfiltration or arbitrary command execution in FortiManager.The cited examples are a GCP Composer dependency-installation flaw and Fortinet advisory CVE-2024-475755.Source: B. Secure Execution of AI-Generated Code · p. 9
V. THREATS TO VALIDITY
- The 51 scenarios sample common threats but do not cover the full spectrum of possible malicious activities.The authors believe the suite captures key concerns within the paper’s scope.Source: §V · p. 9
- Evaluation was limited to Dyff on Linux and a Python implementation; other platforms or programming languages may yield different results.The authors note Linux-based systems and Python are commonly used in ML contexts.Source: §V · p. 9
- Code-LLM-generated candidate tests were not automatically usable, and different models may affect test-case quality.Despite variability, the suite’s core principles are described as broadly applicable to common security and confidentiality issues in untrusted-code execution.Source: §V · p. 9
VI. CONCLUSION
SandboxEval offers 51 scenario-based tests for examining security and confidentiality in LLM assessment sandboxes. Applied to Dyff, its results informed configuration review and offered developers insights into sandbox effectiveness.
VI. CONCLUSION
- SandboxEval comprises 51 malicious-code execution scenarios relevant to assessment frameworks evaluating LLMs for untrusted code.The paper argues assessment frameworks should test whether sandboxing techniques are properly applied.Source: §VI · p. 9
- Dyff’s researchers used SandboxEval results to review deployment details and consider whether additional mitigations were necessary.The suite was applied to a running Dyff instance to assess sandbox security and confidentiality for untrusted code execution.Source: §VI · p. 9
- By probing interactions with system information, structures, contents, privileges, and other factors, the suite provides insights into sandbox effectiveness that can inform safety measures.The authors state this can help developers mitigate risks associated with untrusted-code execution.Source: §VI · p. 9